Forum Time

[Olaf Hart]

Thanks Beau, it starts to gel by the third reading of your post and Jamies one.

So if he started out as a systems architect and is now promoting Devops, the first question is how do organisations deal with the Human Resources consequences of this shift, does it need retraining or replacement, and how do you sell it to IT workers?

[Jamie]

Thanks Beau! You cleared up a bunch of things I hadn't understood.

[BeauV]

Thanks Beau, it starts to gel by the third reading of your post and Jamies one.

So if he started out as a systems architect and is now promoting Devops, the first question is how do organisations deal with the Human Resources consequences of this shift, does it need retraining or replacement, and how do you sell it to IT workers?

I need to sleep now, but I’ll get back to this. It’s a great question. More to come.

[kimbottles]

Thanks Beau, it starts to gel by the third reading of your post and Jamies one.

So if he started out as a systems architect and is now promoting Devops, the first question is how do organisations deal with the Human Resources consequences of this shift, does it need retraining or replacement, and how do you sell it to IT workers?

I need to sleep now, but I’ll get back to this. It’s a great question. More to come.

Beau sleeps? Who knew?

[BeauV]

Thanks Beau, it starts to gel by the third reading of your post and Jamies one.

So if he started out as a systems architect and is now promoting Devops, the first question is how do organisations deal with the Human Resources consequences of this shift, does it need retraining or replacement, and how do you sell it to IT workers?

I need to sleep now, but I’ll get back to this. It’s a great question. More to come.

LoL!! Yes, Kim, I do sleep. But I have to say that after two extended periods at sea (3 and 5 years) standing watch-on-watch, raising two kids (night diaper duty), and now two grand kids (ditto), combined with an ability to wake and fall asleep quickly, I'm generally up and around ever four hours. Thankfully, my lovely Admiral isn't irritated by the old fart prowling around the place from 3-4 am most nights. I have been able to get to know the local skunks, who like that time of night.

Now back to the question.

The primary motivation for DevOps is to break down the traditional barrier between Application Developers (Software engineers who write code specifically for use in operating a company as opposed to Systems Engineers who write code to build infrastructure upon which applications will run.) and IT professionals or Ops folks (Who are the poor bastards who have to keep the infrastructure running and also help keep the Application running). This barrier has been wide and tall for decades. Ops folks complain that AppDevs just write-it, launch-it, and forget-it. (There is some truth to this.) AppDevs complain that Ops folks just whine all the time because they do things like upgrade the infrastructure and break the App. "No my fault that they changed the OS and broke the App." (There is some truth to this.)

So DevOps is a specific attempt to make a blended organization which will think about BOTH issues all the time. The problem is, I don't think this is working very well. But, that's a much longer story.

The first thing we'd have to establish is what level "system" this person is talking about when they say that they are a "systems architect". If it is the architecture of the infrastructure, meaning the OS, the File System, Database, Network, etc..., then it's a long long way from that to being the systems architect of a platform which directly support applications. To help with this, I've attached a picture of a massively over-simplified "stack" of the various parts of a modern computing platform for applications. The complete diagram would fill many pages, but there's no need for that. The answer to your HR question lies at this level of detail.



Here goes:

First, on the right are the areas of the stack that are generally developed and deployed by Systems Programmers and Application Programmers. You'll also note that on the left the Ops Folks are tasked with covering almost the entire stack, at least from an operational point of view; meaning they get yelled at when it's not working. At the very bottom is the muck of hardware which almost no sane company fools around with. Only companies the size of the GAM (Google Amazon Microsoft) actually design their own hardware to run datacenter. We call what the GAM are doing loosely "The Cloud". One last note of overview, note that I've put diagonal lines through the layers from hardware up to Containers or Virtual Machines. The Cloud as delivered by the GAM includes all of this from both a Software and an Operations point of view, leaving the various companies (think: Ford, GE, Goldman Sachs) to free themselves of the personnel and asset expenditures of all the layers below the top of the Container or VM layer.

Second, note that folks who write Applications are really implementing either User Facing code, what you see when you go to a company's web site or their order entry page; or something called Business Logic, which is the sort of code that takes an order from a customer and routs it to a database which is then accessed by the folks in the warehouse as they ready the order for shipment. The User Facing code is designed to let non-tech folks search stuff, read about stuff, etc.... we all do that a lot. The Business Logic code is the purview of a guy like Jamie who wants to insure that the left-handed screwdriver I ordered gets put in a box and mailed to me before I call him up and scream at him. He also wants to measure and audit everything so he can tune up the operations of the company. Putting Business Logic into software has long served as a way in imposing controls on organizations without having humans with clipboard looking over everyone's shoulder.

Third, because Business Logic software has become the control layer for most companies and the actual mode of production for some (think: music streaming or selling ads online) it is only fair that this sort of software and its operation be managed and controlled in a highly disciplined manner. Programmers don't get to screw around with production code. It's also clear that the stuff below the line between Runtime Management and Containers and VMs needs to be rock solid and stable. In traditional IT, that stuff is generally provide by a company like Oracle, IBM, etc... as hardware with dedicated software. In The Cloud, all that and the operational folks to run it are provided by the GAM.

Finally, depending on how one has chosen to implement all this, there will be Ops folks who either keep the top few layers running (here DevOps works better) or there will be Ops folks who run everything from the Sun serves purchased from Oracle up to the web servers and all the associate hardware. Those Ops Folks shown on the left have to know how to RUN all this stuff. They do not need to know how to write the code or the various theoretical bases upon which it rests.

Now, to answer your question directly, the migration from Systems Architect to DevOps discussed here is a migration from the work of architecting and writing applications and their runtime (IMHO) to the work of both developing that code and operating it. This blend has significant problems. Developing is fraught with creative risk, you have to try new stuff and you have to work with Finance and/or Manufacturing and/or Operations to get that new stuff working correctly for everyone (including the poor customer who we maliciously call a "user".) In stark contrast the Ops function is measured on stability and up-time. Creativity have very little value in Ops, for good reason.

I think it would be quite difficult to have someone move back and forth from Systems Architecture (of Applications) to Operational Management. The metrics, approach, attitude, and goals are quite different. For this reason DevOps has generally only worked well when the entire team is small and knows each other well personally.

Sorry for the long answer... it's a complex space.

[SemiSalt]

Just as a comment, anyone stuck in Beau's diagram needs a detailed knowledge of his own layer, and a working "how-to" knowledge of the layer above and below. He will also need a few key facts about the layers above and below those.

[kdh]

I struggle to find the appeal in using Amazon Web Services or the like. We have a little room with a good air conditioner and a bunch of rack-mounted Linux boxes and find it trivial to maintain. We have more fast local memory and a faster network than we can find in the cloud.

Our investment people write code and manage operations. As a technologist I find the idea of hiring someone to yell at when their computer does something unexpected repulsive.

[Olaf Hart]

Wow, thanks again Beau.
It’s another world, but vaguely recognisable after the third reading.
I get the HR implications, it’s the same as my last role resurrecting the PCP training program, I split the job up into two when I handed it on, one doing organisational management, the other delivering education programs.
The two tasks needed entirely different skill sets and mind sets, it was difficult to switch from one to the other many times a day, and I suspect neither was done as well as they could be.
I am leaving my mentor role at the end of this year, the Program now rocks, best pass rates in the country, we chair all the national consultative bodies, knocking back great job and program applicants.
And the Rural Generalist Program is also humming, the feds just appointed a national rural health commisssioner, part of whose job is to cement this program into the national medical workforce.
It is great to have an opportunity to become an uber delegator in ones retiring years.

[Orestes Munn]

I struggle to find the appeal in using Amazon Web Services or the like. We have a little room with a good air conditioner and a bunch of rack-mounted Linux boxes and find it trivial to maintain. We have more fast local memory and a faster network than we can find in the cloud.

Ever thought of mining BTC?

[Ajax]

I struggle to find the appeal in using Amazon Web Services or the like. We have a little room with a good air conditioner and a bunch of rack-mounted Linux boxes and find it trivial to maintain. We have more fast local memory and a faster network than we can find in the cloud.

Ever thought of mining BTC?

Pffft... is Keith, Russia?

You realize that BTC is now at the point where it takes enormous corporate or gov't computing resources to mine a single coin? The days of "Joe Cubicle Farmer" mining bitcoin is over.

[Orestes Munn]

I struggle to find the appeal in using Amazon Web Services or the like. We have a little room with a good air conditioner and a bunch of rack-mounted Linux boxes and find it trivial to maintain. We have more fast local memory and a faster network than we can find in the cloud.

Ever thought of mining BTC?

Pffft... is Keith, Russia?

You realize that BTC is now at the point where it takes enormous corporate or gov't computing resources to mine a single coin? The days of "Joe Cubicle Farmer" mining bitcoin is over.

I was simply making a weak joke about Keith’s server farm.

[Jamie]

Beau, when I read you talking about sleep, I’m reminded of the middle -age habit of first and second sleep, and the period in-between, dorveille where in the quiet period in between, people would get their most creative work done.


For a small-to-medium sized web-based business, the economics of the cloud are pretty hard to ignore.

It’s fast, for practical porpoises infinitely scalable and redundant. We stand up about 7 thousand ecommerce sites, manage the same number of unique product catalogs, have applications in about 19 thousand clinics and process maybe 2-3 million transactions a year and a similar number of physical fulfillments.

It would cost us too much to duplicate the scalability and redundancy of the cloud the same way in-house. The cloud, particularly the Google cloud, also offers a lot of powerful development and connectivity tools at a very low cost. Our stack is all custom-coded open source. Implementations of new applications on the cloud takes half the time because there is no hardware to stand up - it’s already stood up. All I need to do is make sure I have a big enough pipe for the few things left in-house.

[kdh]

I struggle to find the appeal in using Amazon Web Services or the like. We have a little room with a good air conditioner and a bunch of rack-mounted Linux boxes and find it trivial to maintain. We have more fast local memory and a faster network than we can find in the cloud.

Ever thought of mining BTC?

Pffft... is Keith, Russia?

You realize that BTC is now at the point where it takes enormous corporate or gov't computing resources to mine a single coin? The days of "Joe Cubicle Farmer" mining bitcoin is over.

вы обнаружили истину товарища

[kdh]

For a small-to-medium sized web-based business, the economics of the cloud are pretty hard to ignore.

It’s fast, for practical porpoises infinitely scalable and redundant. We stand up about 7 thousand ecommerce sites, manage the same number of unique product catalogs, have applications in about 19 thousand clinics and process maybe 2-3 million transactions a year and a similar number of physical fulfillments.

It would cost us too much to duplicate the scalability and redundancy of the cloud the same way in-house. The cloud, particularly the Google cloud, also offers a lot of powerful development and connectivity tools at a very low cost. Our stack is all custom-coded open source. Implementations of new applications on the cloud takes half the time because there is no hardware to stand up - it’s already stood up. All I need to do is make sure I have a big enough pipe for the few things left in-house.

I suppose if the application interfaces live on the web it makes more sense. For us, getting data to the cloud and results back again is impractically slow.

Also, if we could specify a virtual machine that went beyond the underlying hardware configurations it would make more sense to us. For example, "give me a 128-processor machine with 16T of local memory." This is not possible on Amazon's services.

[Jamie]

For a small-to-medium sized web-based business, the economics of the cloud are pretty hard to ignore.

It’s fast, for practical porpoises infinitely scalable and redundant. We stand up about 7 thousand ecommerce sites, manage the same number of unique product catalogs, have applications in about 19 thousand clinics and process maybe 2-3 million transactions a year and a similar number of physical fulfillments.

It would cost us too much to duplicate the scalability and redundancy of the cloud the same way in-house. The cloud, particularly the Google cloud, also offers a lot of powerful development and connectivity tools at a very low cost. Our stack is all custom-coded open source. Implementations of new applications on the cloud takes half the time because there is no hardware to stand up - it’s already stood up. All I need to do is make sure I have a big enough pipe for the few things left in-house.

I suppose if the application interfaces live on the web it makes more sense. For us, getting data to the cloud and results back again is impractically slow.

Also, if we could specify a virtual machine that went beyond the underlying hardware configurations it would make more sense to us. For example, "give me a 128-processor machine with 16T of local memory." This is not possible on Amazon's services.

We had some applications on Amazon, but found the cloud product and support inferior to Google. Our business starts out on the web, but do a workflow routing of prescription and pharmacy review, so not all of it is web based. The Google cloud is all fully customizable to exactly what you want from performance to virtual machine types. So you can ask for a specific type of hardware and service level. The nice thing is that if demand changes, they automatically apply more resources so you never see a performance drop.

[kdh]

Thanks, Jamie. I took a look at Google and there are some virtual machine options that surpass what we have here. We have some hacks that we use to give us multi-process shared memory in R that might present problems, but at this point it seems we should investigate this at the next hardware upgrade.

[BeauV]

Great discussion guys. You're right on the nub of the issue here.

Once a customer reaches the point where they have a workload (like R for Keith's company) and a specific data set size and performance requirement, then the can balance those needs against the cost of building their own infrastructure below the workload vs using a outsource provider (AKA: "The Cloud")

Factors that folks normally don't include in their calculation include:

- Cost of experts who know how to run all the layers of infrastructure. For the initial deployments, upgrades, and introductions of new technologies. Education and re-education of infrastructure tech staff turns out to be a MUCH larger cost than most folks initially believe.

- Depreciation timeline vs actual useful life of the infrastructure. Most folks expect IT hardware (especially) to be "useful" for many more years than it actually is. When running on The Cloud, the provider is on the hook for unexpected tech obsolescence. To put this in perspective, I know from personal conversation with Microsoft and Google that each of their data centers is different. The technology is moving so fast that by the time they build another data center it's time to redesign the hardware and low-level software infrastructure. Of course, the customers never see that, they are operating up at the Container and Virtual Machine layer and only recognize the change because various data centers perform differently. Many customers use a 3 year straight line depreciation on their own computing hardware. That is clearly absurd in a world where machines double in speed each 2 years.

- Reality of data migration and transfer. Keith nailed it when he said that the data sets he uses are too large to meet the speed requirements for his business. Most folks don't realize that a semi-truck rig is the highest bandwidth way of moving large amounts of data to/from a remote datacenter. A 747 is the highest bandwidth way to move large data sets to/from Asia. Latency on a 747 sucks, measured at about 20 hours, but once the data arrives it's amazing what the actual bandwidth was.

- Data and algorithm security. Many folks (eg: Keith) do things with data in software. These things are the core differentiation of their business and they do not want anyone able to watch. While folks have become more comfortable with the integrity of the GAM (Google/Amazon/Microsoft). It is still a genuine worry.

[Jamie]

We collect transaction data on 19k accounts from 20+ flavors of PIM and put it all on the cloud for analysis. We do it faster than what we could in the past. I'm in Tulsa today talking to our SysOps. He's telling me we just moved a number of large DBs and he's never seen such connection speeds. His other point was, if his back up has a memory issue, while memory is cheap, in the cloud 5min later he has the memory. We couldn't keep up with the CAPEX required to stay at the front.

[Jamie]

Hmmm...Interesting video on machine learning

[youtube]http://www.youtube.com/watch?v=R9OHn5ZF4Uo[/youtube]

https://www.youtube.com/watch?v=R9OHn5ZF4Uo&t=3s

[SemiSalt]

Microsoft has been busy pushing out a major update for Windows 10. On my computer, the update failed with a message telling me to uninstall Trend Micro 2007 and Trend Micro 2009. I don't remember using Trend Micro (though I might have) and the computer is only three years old. Nothing Trend Micro is installed on the PC and the TM removal tool did nothing. There were some hints that some TM technology is borrowed in third part products, and may that could be the problem. I railed that MS could complain about something with telling me where the offending file(s) was.

Tonight I found a post on a MS TechNet showing where in the Registry the offending file is named. I looked there and found a pointer to files in a download directory. I deleted them and tried again. Failure, but now the registry entry pointed to a different copy (due to a backup made during a previous problem) of the same file, and deleted that. After that, success...if the new version of Win 10 can be considered a success. The file were on this computer because the the Documents Folders had been copied from the previous machine.

I really don't know how people who are only users of programs, and who know nothing about internal mysteries like the Registry, are supposed to cope. This took me a couple hours of searching over several days during which I went up blind alleys before I found the road to redemption.

[floating dutchman]

Semi, I'm a "user of programs" as you say. I've found if you just leave things for a while they often fix them self.
If it happens to my machine the it happens to others too, and MS want my machine upgraded, so they find a way.
And if the upgrade doesn't install, then I couldn't care less. None of my computers are worth much (in a $$ sense) and everything even mildly important is well backed up, I even have a portable hard drive I keep at work.
also I keep ISO images of every computer so fresh install is possible too.

Jeroen.

[Ish]

A friend has been having problems with one of his laptops, so I volunteered to help. I finally traced the problem down to a dying hard drive. Put in a new hard drive and it was better, but still not the way it should be. Along the way, I have installed and configured Windows 10 at least fifteen times, and I'm going to keep doing it until it comes out right. Strangely enough, the remaining problem is that it runs all programs very slowly because the CPU is running anywhere from 50% to 99% of capacity even when the computer isn't doing anything. My desktop computer normally runs at around 5% of capacity.
Some of my earlier installs fixed the speed problem but the holes in the hard drive torpedoed it prematurely. I find it odd that the installations would vary so much from one to the next. Thanks, Microsoft.

[TheOffice]

My office computer spent 3 hours doing an update. It ran like crap. I shut it down and restarted. It ran scan disk and fixed itself.

[BeauV]

My office computer spent 3 hours doing an update. It ran like crap. I shut it down and restarted. It ran scan disk and fixed itself.

I think we'll see things like this more and more. Hopefully, it won't require 3 hours of pain and a re-boot. Software is getting better. I know that maybe hard to believe for many of you reading this, but it's true. We're now building software that is self-correcting without things like a re-boot.

It all started with error correcting codes (ECC) in memories (both RAM and disk storage), then moved into debug bit chains within processors and ECC within the CPU chip's memories. Similarly, ECC and re-tries have existed in the network for decades. Now it's moving up to the operating system and clusters of operating systems. Eventually, these machines will have a sort of electronic immune system. Perhaps sooner than we think.

[kdh]

Is there a compelling reason other than security to even bother with updates on a Windows box?

I run Windows 7 on two boxes at home that are behind a hardware firewall with updates disabled. We're careful about email and where we browse. Both have been rock solid.

[Orestes Munn]

This is security related, but it took several generations of updates to finally get my Mac desktop to work reliably with the card reader I’m required to use at work: no read-no work. I probably wasted six critical, morning hours fiddling superstitiously with the damn thing last year. Of course, some things probably got worse...

[TheOffice]

Beau windows NT was a huge step forward in os stability. As an end user and occasional systems administrator everything since it has been window dressing. KDH windows updates are about security. Even with a firewall you are using an unsupported version of Windows. If the firewall fails you are wide open.

[Ajax]

Sorry, but it *is* difficult to believe that software is getting better. 30 years later, and you've still got people on this forum complaining about the exact same problems.
Things are "different" but not much "better." Now that computers are ubiquitous and essential, vendors have even less incentive to make things better. A large focus is on security... and I certainly can't blame vendors for that.

[Orestes Munn]

I am told by our IT people that we are on the receiving end of over 100 malicious attacks a day. I guess it’s hard for that kind of security to be invisible to the user.

[kdh]

The original Windows up to 3.1 or whatever is was was a compete piece of crap.

I could not care less about running an unsupported version of Windows. What I care about is that it works.

Reading the news, phishing attacks seem to be the only real risk.

OM, we see the same thing at our firewall at work. But "malicious attacks" are basically automated attempts to connect in some way, easily thwarted. These should not be taken personally.

There is a large industry associated with computer security. Plenty of hype and technospeak to go along with it.