Thursday, 23 April 2015

Photos (Part 2) - Exchange / Active Directory/ Office 365 Photo Importer


With Part 1 of this series I introduced you to a tool that will allow you to convert photos into a format suitable for importing into Active Directory, Exchange, or Exchange Online. These photos are used across the whole Office product line, including Lync/Skype for Business, Exchange, Sharepoint, Office 365, etc. After creating these images, the next step is to import them into either Active Directory and/or Exchange 2013. The final outcome is having glorious photos appear in your Office applications!

Look! Pretty Photos!

So once again, to try and save everyone a lot of pain, I've made a tool that will hopefully make importing these images a breeze.


Exchange / AD / O365 Photo Importer

The aim of this tool is to be simple and flexible as possible to take the pain out of importing photos in any scenario.


  • View previously uploaded Exchange HD images, Active Directory images, and Office 365 Exchange Online HD images.
  • Import Exchange HD images, Active Directory images, and Office 365 Exchange Online HD images.
  • Remove images from Active Directory or Exchange HD photo for any user.
  • Downscale previously imported Exchange HD images to 96x96 sized images in Active Directory by pressing the “Use Existing HD” button. This replaces the 64x64 image that Exchange auto-uploads to AD when you do a HD image import. This button can be handy if you no longer have the source image on hand and want to quickly upgrade the resolution of your AD photos.
  • Automatic detection of On Premises or Office 365. The system type that is detected when the tool boots and the system type will be shown in the top right hand side of the interface. Note: The system type will affect the naming convention used for the user names in the tool. On Premises will use the SAMAccountName and Office 365 will use the Alias/Username of the user.
  • Automatic resizing of images to 96x96 before they are being imported into Active Directory. This stops you from uploading unnecessarily large images into Active Directory. (ie. if you open a 4MB picture into the tool and try to import it into AD, the tool will convert the image to 96x96 before uploading it)
  • The View Web Image button will open a browser connection to the 648x648 sized version of the image. This can be useful if you want to download a copy of the HD image as a backup.


Requirements:
  • The script is supported on Powershell Version 3.0 and above. So if you're running Windows 7 you will need to make sure you've upgraded your Powershell version to at least Version 3.0.
  • Drag and Drop only works when the filesystem and Powershell session have the same security level. So if you're running Powershell with Administrator privledges (ie. Run as Administrator), whilst you are logged into the machine as a different username, the Drag and Drop function will not work. To fix this just run the Powershell with the privileges of the user you logged in as (as long as you have the correct AD and Exchange permissions) and it will work.
  • In order to set Exchange HD photos for On Premises or O365 the user that is running the Powershell session will need to have permissions to run the Set-UserPhoto command. The build in RBAC roles that support this command include Organization Manager, Recipient Management and Help Desk. To set Active Directory photos the user will need permissions to run the Set-AdUser command.

Version 1.01 Update (15/5/2015)

  • Corrected issue with the tool on Powershell version 4. Removed "-ErrorVariable" flag from script because it was causing "language mode" errors on Powershell Version 4 with Remote Powershell connections.





Overview


The tool is designed to gracefully fall back to support whatever level of Powershell commands that are available to it. So if you were to run it on a machine that only has access to Active Directory commands it would only allow you to import Active Directory photos, and so on. Ideally you should have access to both Active Directory and Exchange commands for an on premise deployment. For Office 365 you will need to remotely connect to Exchange Online (see the next section for details of how to do this).

There are a few things that you should understand about how the Powershell import commands work before using the tool. The Exchange import command (Set-UserPhoto) supports the importing of any sized file JPG file into the system. If the file is not square in shape then Exchange will do a “centre crop” (as explained in my previous post) on the image and convert it to 648x648 in size then import it into the user’s mailbox. At the same time as doing this Exchange will also import a 64x64 sized image into Active Directory.

The Active Directory import command (Set-ADUser -identity $name -Replace @{thumbnailPhoto=$photoBytes}) does not support the same fancy cropping and resizing capability as the Exchange command does. Instead it will import the raw bytes that it is presented into the thumbnailPhoto attribute in the Active Directory database. The thumbnailPhoto attribute will accept images of up to 100KB in size. However, it’s not recommended to import files that are that big into Active Directory as it can add a great deal of size to the database which can result in much larger amounts of replication traffic between Domain Controllers. If you import an image into Active Directory and there isn't currently a HD image in exchange, then the AD image will also be displayed by the Get-UserPhoto command in exchange as well.


Tool Operation


Now that you have an overview of how the commands work we can go into some depth about how the tool works. The tool displays three images, the left most image (Input Image) is a preview of the image that you are going to import into the system. You can select this image by either dragging and dropping the image from your PC (see requirements section for more details of this), or by selecting the browse button under the Import Image section. The middle photo is the current Active Directory photofor the user highlighted in the Select User dropdown box. The rightmost photo is the current Exchange HD photo for the user highlighted in the Select User dropdown box. If the user does not have images in either of these locations then a generic missing photo image will be displayed by the tool. If the tool has not been able to access the necessary commands to get to the photo a “Not Accessible” message will be displayed. If the user does not have a Mailbox a "No Mailbox" image will be displayed.

When using the tool you may choose to import a single user photo or you might want to import a folder full of images. The tool will allow you to do both by selecting the Import Folder or Import Image checkboxes respectively. When the tool opens a file it will automatically select the import option appropriate for the image size. If the photo is 96x96 or smaller, then the Active Directory import check box (Replace 96x96) will be selected. However, if the image is larger than 96x96 then the Active Directory (Replace 96x96) and Exchange (Replace 648x648) import checkboxes will both be selected. In the case that both of the checkboxes are selected the tool will first import the image into Exchange (which will automatically import a 64x64 image into Active Directory) and then the tool will resize and import a 96x96 version of the image into Active Directory (ie. over the top of the smaller 64x64 image). 

When you select to import a folder you have the choice of seeing each image as it is imported and selecting whether you want to import the image or not. This by default is the operation of the tool, however, you may wish to import all of the photos without confirming each file. This is done by unticking the Confirm Import checkbox. When importing folders, the files within the folder need to start with a name that matches the user name in Select User dropdown box (which is the user's SAMAccountName from Active Directory for On Premises, or the Alias/Username from Office 365). The tool will allow you to have extra information in the file name, however, additional information must be separated from the user's name by a space character (space is used because it can’t be used in a SAMAccount name or Alias/Usernames in Office 365). For example, you may have a file named “John.Smith 648x648.jpg” which the tool will import for a user with the SAMAccount name of “John.Smith”. However, you cannot have the name “John.Smith648x648.jpg” because it doesn't have a space character between the name and additional test which means it isn't an exact match for the John.Smith user in the dropdown box. The important point to take from this is that you need to be precise in the naming of your images for a Bulk import. In summary:

On Premisis File Naming: SAMAccountName
Office 365 File Naming: Alias/Username

Note: Image file names must begin with the user's name following the convention above and must be divided from any other text in the file name by a space character.

The file naming for a bulk import is your most important job! After doing this you can kick back and let the tool do its work.

Importing Photos into Office 365


The tool has been designed to support importing HD photos into Exchange Online. However, the commands used to connect to Office 365 via Powershell may be slightly different than what you usually use. So Office 365 admins - pay careful attention to this section!

Create an O365 session:

$Session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri https://outlook.office365.com/powershell-liveid/?proxymethod=rps -Credential $cred -Authentication Basic -AllowRedirection

Note: You must include the “?proxymethod=rps” part of the URI in order to import images into Exchange Online. If you don’t do this, the import will fail with a (413) Request Entity Too Large error.

Enter O365 credentials:



Import the session into Powershell:

Import-PSSession $Session

After you have done this, you can run the script in the same Powershell window and import photos.



Limits of Office 365


There does appear to be a maximum image file size that you can upload to O365. However, it seems to be more of a practical size limit rather than an enforced one. I have uploaded images up to about 6MB in size before, however this is a very slow process and you can run into random errors during the process. It can also take a long time (5-10 minutes) to import very large images directly into the Exchange Online server. When the photo has been uploaded the Exchange server will resize the image before saving it to the user’s mailbox. For the aforementioned reasons I would recommend reducing the size of this photo before trying to upload to Exchange Online. My suggestion is that you use my Image Creator Tool to reduce the size of your photos to 648x648 before you try to upload them.

I have found that Office 365 doesn't like you uploading photos for the same user multiple times in a row (ie. a bulk import with many photos for the same user name). If you do this you will get errors back from O365. However, if you are bulk uploading to different users it seems to work fine. If you start experiencing random errors when trying to upload or get images from Office 365 try disconnecting the current session and open a fresh Powershell session.

On occasion when uploading an image to O365 the upload process will work successfully (no errors in the PS window), however, the image will not be available to be viewed immediately after a successful upload. Majority of the time this is not the case but occasionally it will take about 10 seconds before the photo can be viewed. In this time the tool will display the generic no photo silhouette picture. If this happens, be patient and refresh the user photo by reselecting the user from the 'Select User' drop down box. 


The Wrap Up


Well, that’s photos in the bag! You should now be able to convert and create images as well as import them into an On Premises system or an Office 365 tenant. Hopefully this makes your life much easier. Let me know if you have feedback and enjoy!



Read more →

Friday, 13 March 2015

The Case of the Lync 2013 Edge Server Centralised Logging Ports

If you have ever done any port testing on a Lync 2013 Edge server you may have noticed that the external interface of the Edge server had ports in the 50001-50003 range open and listening for TCP connections. Usually this is not the case in the 50000-59999 media range because these ports are only opened for short periods of time when media ports have been allocated by the Edge server for active calls. The Media Relay Service on the Edge was designed this way for security purposes, so that the 50000 range could be opened on external firewalls without posing a significant security threat. So what are the undocumented 50001-50003 ports facing externally? Well, they are actually the Centralised Logging Service and they appear to be facing externally for no reason other than they are bound to port 0.0.0.0. The netstat command show this:

ClsAgent.exe Ports


The netstat output shown above shows that the ClsAgent.exe service is listening on the 50001, 50002 and 50003 ports on the IP Address “0.0.0.0”. This means that the service has not bound to any specific interface and as a result will listen on all interfaces. I image this is a design issue with the CLSAgent because it was originally designed to run on internal Lync servers that only had one interface and so binding to a specific port wasn’t a requirement at the time of designing the software. I’m happy to be told different by someone at Microsoft though…

As mentioned earlier, the CLS port range falls into the TCP 50000-59999 range, which is also legitimately used for Edge Media Relay service. So it is included on the list of external port ranges that may be open on the external firewall. I say may be opened because this range of TCP ports does not need to be opened inbound (as per the guidance from Microsoft) unless you are federating to OCS 2007, or in the more complex scenario when you are using NATing with DNSLB on the external edge and your firewalls do not support hairpinning (ie. traffic coming from one Edge server’s NATed external public IP address back in to another Edge servers NATed External Public IP address) of media between multiple edge servers in the same pool. Another legitimate reason for this is when you want an optimised media path that does not require tunnelling via port 443/3478 to get to the required 50000 range media port. These scenarios were explained in a great amount of depth by Bryan Nyce and Thomas Binder at Lync Conference 2014. I suggest you watch these videos several times if you don’t understand what I’m talking about here.

'So what percentage of companies actually open the 50000 range of ports to the internet?' I hear you asking… Well, I also wondered this, so I decided that I might do some research and find out. I tested approximately 250 Edge servers of some of the largest organisations in the world and found that approximately a quarter of them have the 50000 range open (with CLS ports showing). So quite a large number of organisations currently have this issue.

The actual security implications of having these ports open on the Internet is not fully known at this stage. The existing ClsController.exe application and CsCls Powershell commands supplied with Lync do not allow the user to connect to servers outside of the Lync pools within their installation. So it's certainly not the case that you can use them to randomly connect to other organisations' Edge servers and start logging service. It's my understanding that Microsoft is aware of this issue and have not yet done anything to change the behaviour, so they obviously have deemed it low risk. In my opinion though, the reduction of attack surface is always a good idea for internet facing services. So my recommendation is to block these ports because they serve no practical function externally.


A Work Around


I have written a Powershell script that will block the CLS service ports on selected interfaces of your Edge server. This will function as a work around until Microsoft decides to formally change this behaviour in the product.

When you run the script it will display a list of IP Addresses on the server that you can choose to block the CLS service on. You simply need to enter the number of the interfaces in the list that you would like to block access to the CLS ports on. The script will then automatically create a new firewall rule to block TCP ports 50001-50003 inbound on the selected IP Address to the CLSAgent service on the machine. Follow this process for all externally facing Edge IP Addresses (ie. Access Edge, AV Edge and Conferencing Edge IP Addresses).

BlockCLSExternalEdgePorts1.00

The rules that are added can be seen in the Advanced settings of Windows Firewall in the system Control Panel:



Once you have done this for all of your Internet facing Edge IP Addresses you can rest easy: your Edge is now as safe as you previously thought it was…




Note: Run Powershell as Administrator when running the script.

The Wrap Up


Security is important and we all need to try to understand what we are asking when we tell the firewall team to open internet facing firewall ports. Whether you are deploying a new Edge server or already have an Edge server out in the wild, then I suggest you implement my firewall work around. Enjoy and see you next time. Ciao!



Read more →

Tuesday, 17 February 2015

Photos (Part 1) - Lync / Skype for Business Photo Editor

Happy New Year! Hang on, it’s February… Time seems to have got away on me since I started this holiday project that turned out to be more complex than I had initially expected. What was originally going to be a simple photo resizer tool for Active Directory, Exchange and Lync/Skype for Business, turned into more of a complete photo editor including face recognition based cropping, image filtering and processing, written entirely in custom Powershell code… What can I say, sometimes I can’t help myself…

So let’s begin by discussing the problem: Lync and Skype for Business clients are used for communicating, and the best experience for this is when you are able to easily see the people that you’re communicating with. So within the Lync and Skype for Business client UI there are many places where photos of users are shown. This is great because it makes the software much more personal and approachable to users... and ever since Bill Gates' mug shot silhouette was replaced as being the default missing photo image from Lync 2010 (yes, is really was, see below!) to the generic Lync 2013 oval head silhouette, a photoless client tends to look a bit boring.

Image Reference

So for the best user experience we really should be using profile photos for users in Lync and Skype for Business… but what does that mean for us Lync Administrators? Well, usually it means that you are going to get given thousands of random image files, in god knows what format, and at who knows what quality, that you will be expected to magic into the system and have displayed for all to see. So let us put our magicians' hats on and I will help you with Part 1 of this trick… Turning a mess of image files into something that will work when imported into Active Directory and/or Exchange for display within Lync or Skype for Business…

Lync / Skype for Business Photo Editor


You will know by now, if you’ve ever read my blog, that a custom written Powershell tool will always provide the solution to our problems! I give you the Lync / Skype for Business Photo Editor Tool!



Features:
  • Zero installation.
  • Signed Powershell Script.
  • Bulk conversion of a folder full of images files.
  • Custom coded image processing!
  • Manual editing and cropping of individual files. Simply Drag and Drop an image into the Picture Box area and then start selecting the crop box size and position you would like your output files to be based on. Set filter options and preview their effects on the image using the “Preview Filter” button. The size of the image crop box is shown under the scaling tools to allow you to know if you are cropping to a size smaller than image you are wanting to output (to avoid accidently upwards rescaling of the images).
  • Accepts input files in the following formats: ".jpg", ".jpeg", ".gif", ".png", ".bmp" and ".tif”. All files get converted to “.jpg” format so they can be easily imported into Exchange/AD.
  • Smart Crop / Centre Crop modes – By ticking this box (default) my Smart Cropping algorithm will be used to discover the subject's face and crop appropriately. The Margin setting is used in conjunction with Smart Crop to determine how loose or tight the framing will be around the subject. If unticked, a simple Centre Crop Method will be used. See the Smart Crop section for more details.
  • Filters! Since Instagram became such a big hit, filters have become a must-have for all successful software projects. So why should this one be any different? Photo filters include: Colourise (Blue, Red, Green, Yellow, Orange, Pink, Purple), Contrast (Reduce Contrast, Light Contrast Boost, Medium Contrast Boost, Mega Contrast Boost, Ultra Contrast Boost), Brightness (Reduce Brightness, Light Brightness Boost, Medium Brightness Boost, Mega Brightness Boost, Ultra Contrast Boost), Effects (Old Film, Vignette, Light Leak, Vintage, and Slide Show).
  • Output sizes. By default the tool will output 96x96 and 648x648 sized photos. These can be turned on or off using the checkboxes in the “Image Output Settings” area of the GUI. There is also the option to create custom sized photos by ticking the custom checkbox and selecting the pixel width/height of the photos to be outputted. It is generally recommended that 96x96 files are used for uploading to Active Directory and the 648 x 648 images are uploaded to Exchange 2013.
  • Quality Control – The quality of the jpg images that the tool will output can be changed by reducing the Quality setting between 1-100. I suggest that you never actually reduce this in the process of creating the files that you are importing via Exchange, as the quality will be further reduced by Exchange as part of the import process.

Requirements:

  • The script is supported on Powershell Version 3.0 and above. So if you're running Windows 7 you will need to make sure you've upgraded your Powershell version to at least Version 3.0.
  • Drag and Drop only works when the filesystem and Powershell session have the same security level. So if you're running Powershell with Administrator privledges (ie. Run as Administrator) the Drag and Drop function will not work. To fix this just run the Powershell with regular privileges and it should be okay.

1.01 Update (23/4/2015):

  • Added policing of the folder name to accept ending with a "\" or ending without a "\".
  • Changed the output file name to use space (" ") instead of a minus ("-") character between the name and the image size ("648x648") to work with new Photo Importer Tool.





Features - Smart Crop


One of the main problems with Exchange / AD / Lync images are that they must be square in shape, and digital cameras don’t usually take square photos (they are usually taken in a 4:3 or 3:2 ratio).  So as a result, after taking a photo it inevitably must be cropped before it can be used by Exchange or Active Directory as a Lync / Skype for Business photo. The way that Exchange handles this is to do (what I call) a Centre Crop on the image. This is where you crop to a square that is the width and height of the short side of the photo and then centre the square in the middle of the long edge of the photo (see the Centre Crop example image below). This works well when the image has been composed with the subject's face right in the centre of the frame. However, what if the person taking the photo decided to also include a large portion of the subject’s body in the frame? Or it’s one of the user’s favourite photos of them at the beach with the majority of frame consisting of landscape? Or what if, god forbid, the photographer decided to use their arts degree and frame the subject using the Golden Ratio or the Rule of Thirds… In these cases you can end up with a weird looking image if Centre Cropping is used.

Centre Crop Example

I realised all of this after starting this project and tried to think of a better way… like, what if I was to detect where the subject's face was within the image and then crop around it? That sounds like fun! So I started reading about facial recognition techniques and software. What I learnt was that there is no existing core Dot Net libraries that I could leverage in Powershell that would supply me with facial recognition. So I looked more broadly and discovered that there are a couple of open source projects that had Dot Net ports and could be used if imported and compiled into a Dot Net application… but I wanted this to be a pure Powershell implementation and not some bulky application! So I then dug deeper and started reading academic papers about different face and skin recognition methods that exist. After doing much prototyping and testing with these skin recognition concepts, I created my own skin thresholding algorithm in Powershell that was quite fast (something learned whilst doing this was that Powershell is slow at doing many mathematical operations, and especially slow when it comes to recasting variable types and object creation).

The end result of this absurd amount of work was an unassuming checkbox in the Bulk Import section of the tool’s GUI called “Smart Crop”. Smart Crop is used in two places in the tool, the first is when Bulk converting images from a folder.  The tool will try and locate using my skin recognition algorithm where the majority of skin is on the screen and then try to appropriately frame the around this location. The (3:2 ratio) photo below shows an example of what a better alignment for a centre cropped image would look like as a result of recognising where the face is in the image.

Face Aligned Centre Crop

As you can see in the above photo, the image gets cropped around the more important part of image rather than the centre of the image as seen in the earlier Centre Crop example.
So it’s pretty useful to be able to align a full width crop around the subject's face like this, however, Lync photos end up being reduced to very small sizes in most cases (96 x 96 pixels). So it would also be nice to be able to crop even tighter to the subject's face so that you can see it more clearly in your contacts list in Lync / Skype for Business. Below is an example of a better crop to use in Lync:  

Ideal Smart Crop for Lync

In order make the photo more usable with Lync/Skype for Business, I have tried to tune the algorithm to give a tight crop around the face of the subject in the photo. A pitfall of doing a tight crop like this, however, is that the cropping square should not be less than the size of the image that is being outputted (ie. the crop square being 200 x 200 in size and the output size being 648 x 648 in size). If this happens then the quality of the output photo will be significantly reduced due to the image being blown up. For this reason you should always try and use source images that are quite a bit larger than the largest size image you are trying to output. The tool has been designed to understand this issue and will always attempt to crop to at least the size of the output image file. This in some cases will result in a looser crop that you might expect around the subject's face, however, it is designed to maintain the quality of the output file.  

The “Margin” setting in the tool can also be used to tighten or loosen the Smart Crop frame around the subject's face. This setting ranges between 1-50, with a default of 25. The lower the value, the tighter the crop will be around the subject.

From the testing I've done so far I have found my Smart Crop detection algorithm works in the majority of cases (note: it does not work on greyscale images). However, it can have problems if there are background components in the image that fall into the same Luma and Chroma ranges as skin does. In these cases you can manually crop the individual files that were not detected accurately. In the end though I hope it saves you a bunch of time and effort!


Features - Filters


Photo filters are all the rage at the moment with every social media app in the world jumping on the bandwagon. However, the idea of filters in this application is not just a gimmick: it offers you the ability to do colour, brightness, and contrast correction to photos in order to give them more pop so they look their best when displayed in Lync/Skype for Business. When you are supplied photos by an organisation, it’s fairly likely that they were taken in a room somewhere with bad or at least uninspired lighting. As a result, all of the images can look washed out and flat.

Below is an example of a washed out image of a technology company CEO you may recognise. As you may be able to see, the photo on the left does not look very vibrant and comes across as quite bland (especially when reduced to 96 x 96 pixels in size). However, after applying a contrast and brightness boost filter to the image (as can be seen in the picture on the right) it looks much more dynamic.

  


The Lync / Skype for Business Photo Editor Tool gives you the option to Colourise, Contrast Reduce/Boost, and Brightness Reduce/Boost with various levels of intensity, and these settings can be chained to give you 175 different combinations to use!

In addition to the more subtle image quality and dynamics filtering capabilities I decided that I too couldn’t resist the challenge of implementing some grungy Instagram style filters. So if you have been using Instagram too much and feel the need to make your images look a little more vintage, try the Effects filters drop down box. Here’s some examples of my custom effects filters:


Will these filters ever be used for a Lync or Skype for Business deployment? Maybe not, but I had fun figuring out how to write the image processing code to generate them J


The Wrap Up


Well there you have it: my holiday project has finally made to a public release. You may have also noticed that this post is only Part 1 of a series. Indeed it is! Because now that you have a tool to easily create image files for Exchange and Active Directory, you will likely also need a tool for easily uploading those files to these systems. So Part 2 of this series will supply you with just such a tool… So keep an eye out for that one. Cheers, and enjoy!



Read more →

Popular Posts