Will the Real Hybrid Cloud Please Stand Up?

A long, long time ago, so long ago that the distance back can be measured in years, not weeks or months, Hybrid Cloud meant a deployment model that was a combination of private and public cloud that were bound together.  Private would often mean on premises.  This was part of the definition, as written by NIST:  http://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-145.pdf

However, the term hybrid has been acquired for use to also mean deployment across heterogeneous environments, such as different public cloud providers.  An episode of Enterprise DevOps Initiatives podcast this past summer titled "Hybrid Cloud is Not a Choice, It’s a Realization" (https://www.cloudtp.com/doppler/hybrid-cloud-not-choice-realization/) has a title that seems somewhat provocative when taken in context of the earlier definition.  Private clouds tend to be mandated by a rather specific requirement, such as data location regulations, and are not necessarily a common necessity.  So to state that it is a realization to eventually come to seems somewhat hyperbolic.  However, in the context of the newer definition, the title is much more mainstream. 

Services offered by major public cloud providers such as Amazon, Google and Microsoft have become complex and varied beyond the basic storage, compute and network they started with.   A quick look here: https://cloud.google.com/products/ or here: https://aws.amazon.com/products/ shows this.  So the argument they are making is that in order to make good use of cloud services, you end up using many of the products.  And inevitably, the differences in available products will cause you to customize for one public cloud provider or another.  This will be true even for service products that are equivalent across cloud providers, as there will be some variation in implementation.  According the discussion between the host, Mike Kavis, and the guest, Rob Hirschfeld, this definition of hybrid would extend to differences in software versions of the same product if the version differences required migration problems for the application using it.

One attempt at mitigation of this is to only use the basic cloud services - such as compute, storage and networking mentioned above.  However, doing so loses many of the advantages of a cloud based implementation.

The guest suggests that hybridization cannot be eliminated, but simply must be managed.  The approach advocated, as can be seen at his company RackN https://rackn.com/ and the associated project Digital Rebar http://rebar.digital/ , is to compartmentalize the variations between platforms so that changes required for portability are isolated.

In another episode from a different podcast, Cloud Weekly Podcast, titled "Vendor Lock-In is Unavoidable" (https://www.cloudtp.com/doppler/vendor-lock-unavoidable/) the discussion echoes some of the same ideas.  Namely, that using the comprehensive set of services available from a cloud provider gives great benefits, but also leads to being tied to that provider.  The solution is, again, plan for it and manage it.

Now, before we shed a tear for the passing of the previous definition of hybrid cloud computing, observe that it is still in use.  Last week there was announcement of a partnership between VMware and AWS: https://blogs.vmware.com/vsphere/2016/10/vmware-aws-announce-strategic-partnership.html . Note this sentence from the announcement: "Currently in Technology Preview, VMware Cloud on AWS, will bring VMware’s enterprise class Software-Defined Data Center software to the AWS cloud, and will enable customers to run any application across vSphere and VMware Cloud Foundation based private, public and hybrid cloud environments."

Additionally, the original definition of hybrid cloud becomes more nuanced as we move forward.  Microsoft spent a significant amount of time discussing hybrid cloud at its recent Ignite Conference.  The graphic at the beginning of this article reporting on Microsoft's attention to the subject - http://www.zdnet.com/article/why-microsoft-is-circling-its-hybrid-cloud-wagons/ - shows that the concept of hybrid cloud is being stretched to include public cloud services combined with on premises servers (ex. Hybrid Database = Azure SQL Database + SQL Server; Hybrid Identity = Azure Active Directory + Active Directory).  This interpretation makes sense, of course, for Microsoft as it plays to their established strengths of existing enterprise software.

So, will the real Hybrid Cloud please stand up?

The Bonus OpenSSH – Or Not?

Previously, I connected into the OpenSSH server that became active by default if you activated Developer Mode in Windows 10 and were running a recent enough version.  This, however, connects into a standard Windows cmd shell which is not what I was hoping for.  So, with a goal of connecting to the WSL Bash command line there are a couple of different routes that immediately come to mind.

One route is to attempt an install of openssh-server in the WSL.  The method for this is pretty straightforward.  Root shell and apt-get install openssh-server.  Install starts out with no surprises, packages pulled down, unpacked, keys created.  Right until it attempted to have Upstart kick things off.  That is when we hit the wall.  The relevant error messages spit out during the installation were:

initctl: Unable to connect to Upstart: Failed to connect to socket /com/ubuntu/upstart: Connection refused

runlevel:/var/run/utmp: No such file or directory

invoke-rc.d: policy-rc.d denied execution of restart.

So the services needed to get this executing are not running and are not being started on demand.  This can be further verified by doing a process list, where the only background process running is init, and this init is kicked off with the command /init, not /sbin/init as would be normally expected.  Furthermore, /sbin/init does exist (along with many of the other files that would be expected in /sbin) but it is different than the one running.  Is this because the system calls needed to handle this are not yet implemented?

In a series of videos explaining the WSL (Windows Subsystem for Linux) architecture here:

https://blogs.msdn.microsoft.com/commandline/2016/06/02/learn-more-about-bash-on-ubuntu-on-windows-and-the-windows-subsystem-for-linux/

they talk generally about how Linux processes work with the NT kernel and how support has been implemented.  Which is basically that they have a driver in the kernel that handles Linux system calls.  So, there are two items of note relating to this.  1) In the video on architectural overview, Deepu Thomas mentions that sshd requires a good pseudo terminal implementation, which is not yet done. (Although this particular video was recorded back in April, so you can always hope they have made further progress.)  2) In the video on WSL system calls, Stephen Hufnagel does an estimate of their progress on Linux system call implementations.  He says that out of approximately 300 system calls, they have implemented somewhere in excess of 150.  The rest they are still working on.  So, have they yet implemented the ones required for the particular services we are interested in?  I guess I’ll circle back to that.

Another route to getting a Bash shell via ssh is to go through the existing openssh server.  Since you can run bash.exe from the cmd command line and ssh’ing in drops you into a cmd command line, can you jump to Bash from there?  Attempting to execute bash.exe after connecting with ssh spits out the message:  Error: 0x80070020.  Attempting it again at a different time:  Error: 0x80070005.  Either way, no dice.  A little more investigation leads to a comment in a bug report and a surprise (to me):

Closing this issue for now since, as mentioned above, this SSH server is only a minimal SSH server used for device discovery scenarios and it doesn't support all the high-fidelity SSH features one would expect or rely upon.

If you want to run an OpenSSH Server on Windows, know that the real OpenSSH is currently being ported to support Windows as we type: https://github.com/PowerShell/Win32-OpenSSH

Link bug report: https://github.com/Microsoft/BashOnWindows/issues/777

So, maybe our bonus OpenSSH server is not the bonus previously assumed.  Connecting in via ssh and using the –v flag to get version information (among other things), reveals that the reported version of the default ssh server enabled with the Developer Mode turned on in Windows 10 is “Remote protocol version 2.0, remote software version MS_1.100”.  Huh.  The version reported by an ssh server running on XUbuntu 14.04 gives “Remote protocol version 2.0, remote software version OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.8”.  So, delving into the above GitHub project - releases, wiki and installation instructions - it would appear that the default ssh server is not that one.  Evidently, not quite the assumed bonus.  And we all know what happens when you assume.

The Bonus OpenSSH

So, continuing to determine how far I can get with the Windows Subsystem for Linux (WSL) -  Previously, it was pretty straightforward to clone in a Git repository that resided on another machine running XUbuntu 14.04.  Unfortunately, cloning out a repository living in WSL did not work as well.  The default protocol I used went over ssh and didn’t connect in.  Given that I hadn’t installed sshd yet, this is not entirely surprising.  But I thought perhaps it was installed by default.

 

Poking around trying to find what network services are available showed what is not.

Doing an ifconfig from the bash shell returned the message:

The command could not be located because '/sbin' is not included in the PATH environment variable.

This is most likely caused by the lack of administrative privileges associated with your user account.

ifconfig: command not found

Switching to root and repeating ifconfig yielded a different message, but still not the data being searched for:

Warning: cannot open /proc/net/dev (No such file or directory). Limited output.

Neither was ping available from the bash command line:

ping: icmp open socket: Permission denied

Apparently, network services in WSL are still limited.

So, while testing an ssh connection in from another machine to see if anything was already listening led to an unexpected discovery.  The was an ssh server receiving connections and these connections were going to the standard Windows command line.  It was using the Windows login usernames and credentials but not the usernames from the WSL.  Doing a little online searching revealed the project behind this.  Here is the initial announcement from Microsoft:

https://blogs.msdn.microsoft.com/powershell/2015/06/03/looking-forward-microsoft-support-for-secure-shell-ssh/

And then a later announcement of public availability of the code:

https://blogs.msdn.microsoft.com/powershell/2015/10/19/openssh-for-windows-update/

Note that while it is on by default, it would seem that it is only installed if you enable Developer Mode (Settings->Update & Security->For developers) and also if your OS is up to Version 1607 (for Windows 10 Home edition).  I have a Windows 10 machine still on version 1511 and enabling Developer Mode did not cause OpenSSH to be running. (Neither did it provide for the option of turning on Windows Subsystem for Linux.)

So I guess we’ll have to see what sort of conflict arises by trying to install sshd within the WSL.