Since the introduction of FMA, VDA restart schedules in Citrix Studio have been very limited allowing for one schedule either once weekly or else daily at one specific time with the option to restart half the Delivery Group between two different times.
With XenApp and XenDesktop 7.12 to Virtual Apps and Desktops 7 1808, you have the ability to use PowerShell to create multiple restart schedules. This functionality was added to the Studio console in Virtual Apps and Desktops 7 1811. You can read about how to configure multiple reboot schedules via the GUI here.
If you have not yet upgraded to Virtual Apps and Desktops 7 1811, the method of configuring multiple reboot schedules via PowerShell is great because it gives us back most of the flexibility we had with scheduled restarts of machines back in XenApp 6.5 using Worker Groups for example. You may not want to restart all VDAs in a Delivery Group every night or once weekly, so we can turn to PowerShell to configure it how we want. Another new feature introduced to 7.12 called tag restrictions allows you to splice up Delivery Groups so that only certain machines are restarted during different times.
It is important to understand how restart schedules work in Virtual Apps and Desktops 7 (formerly XenApp and XenDesktop 7).
When you set VDAs in a Delivery Group to restart on a schedule, you specify a start time and a time you want the restart schedule to complete by, for example within 2 hours. This means if you have 20 VDA machines, with a reboot schedule time of 2:00am, all machines should be restarted by 4:00am. What actually happens is the Delivery Group is split into two separate groups, Group1 and Group2 for example. The reboot schedule duration time is also divided by the number of machines in a Delivery Group. In this case, that is 120 minutes divided by 20 which equals to 6 minutes. This means, one VDA is rebooted every 6 minutes. Machines in Group1 are preferred for reboot because they have fewer action sessions, and are not in maintenance mode for example. If you are using tags for reboots (introduced in XenApp/XenDesktop 7.12+) then those machines that have the correct tag are split in to groups and restarted based on calculated preference. When a machine is rebooted, a re-evaluation of the preferred VDA to reboot next is performed so that the best placed VDA for reboot at that time is always chosen.
When a Delivery Group reboot schedule commences, all machines in the first phase are put into drain mode, which prevents new sessions from being launched. Keep in mind that existing sessions are reconnected. Also keep in mind that machines in phase 2 are available for new connections until phase 1 has completed the reboot cycle. When phase 1 is complete, those machines are taken out of drain mode and allowed to accept new connections. At this stage, phase 2 VDAs are placed into drain mode and the same process occurs.
To read up on power managing VDA’s including powering off VDA’s after log off etc. see https://jgspiers.com/xendesktop-xenapp-power-management/
To configure restart schedules, we use the RebootScheduleV2PowerShell cmdlets which are new with XenApp & XenDesktop 7.12. Remember that PowerShell is only required if you have not yet upgraded to Virtual Apps and Desktops 7 1811.
To get started using a Delivery Controller launch PowerShell and add the Citrix PowerShell snap ins using command asnp citrix.*
The Citrix.Broker.Admin.V2 module contains all the cmdlets we need to configure scheduled restarts. The cmdlets with V2 on the end are the new cmdlets we will use to confiure flexible restarts. The original commands (which Studio reads) are Get-BrokerRebootCycle for example. If you create a reboot schedule within the Studio GUI it simply runs the non-V2 cmdlets in the background.
As an example, I have created a simple daily reboot schedule using Citrix Studio. When you run the Get-BrokerRebootSchedule command the schedule job appears including the Delivery Group (DesktopGroupName), the frequency of reboots and scheduled start time etc.
When you run the new cmdlet Get-BrokerRebootScheduleV2 notice my reboot schedule appears here as well but there are some additional parameters that can be used such as RestrictToTag.
For now I will delete the current reboot schedule so we can start fresh.
As mentioned before to get granular with reboot schedules we can use tags. You assign tags to one or more VDA machines, and then create reboot schedules within PowerShell and link the schedules to your tags. Right-click a VDA and select Manage Tags.
No tags have been created yet so click Create.
Enter a name and description then click OK. Your tags could be department tags or anything else that makes it easy for you to understand and identify the machines you are working with.
Click Save.
Now that we have one VDA using the HR tag, I can create a reboot schedule. The command I am using is:
New-BrokerRebootScheduleV2 -DesktopGroupUid 1 -Frequency Weekly -Name WeeklyRebootSchedule-HR -RestrictToTag HR -StartTime 01:00:00 -RebootDuration 0 -WarningDuration 15 -WarningMessage “Please log off as this machine will restart in 15 minutes”
This means any VDA with a tag of HR (VDA1) will restart weekly on a Sunday at 1AM. A warning message will be given 15 minutes before restart. No further warning will be given.
Note: To find out the UID of a Delivery Group run command Get-BrokerDesktopGroup | fl name, uid
Note: Instead of using -DesktopGroupUid you can use -DesktopGroupName
I have added a new machine (VDA2) to the same Delivery Group. This time I will use a different tag so click Create.
Enter a name and description. Click OK.
Click Save. This machine (VDA2) will have a tag of IT.
You can see VDA1 and VDA2 with different tags.
I will now create a reboot schedule for the VDAs carrying an IT tag. This reboot schedule will be daily at 3AM.
When you look in Studio, nothing shows under Restart Schedule because Studio cannot read the V2 cmdlets (xxx-BrokerRebootScheduleV2).
And that is us done. Using command Get-BrokerRebootScheduleV2 both reboot schedules show. VDAs within the same Delivery Group will now have their own restart schedules.
When a machine restarts (reboot schedule in progress) the Active parameter changes to True.
The Citrix Broker Service reports a successful power action was achieved for VDA2 during it’s scheduled reboot.
The GroupReboot for Delivery Group NewWS2012R2 has successfully completed. Notice how only one machine was processed even though there are two machines in the Delivery Group. That is the power of tagging and restart schedules.
Another example use case could be a situation where you do not want to restart all Delivery Group VDAs once per week, and once per day is too often. Maybe you would like to restart every Monday and Friday. This is possible by simply creating two reboot schedules, one for Monday and one for Friday as shown below. I have not used tagging, so these schedules apply to all Delivery Group members.
Holger
April 18, 2017Hi George,
thanks for this detailed article. I have
Holger
April 18, 2017damn… too fast… 😉
I wanted to write the following: I also configured some new Reboot Schedules with different tags. In my cases the reboots only worked with no user logged on.
If any user is logged on they will get the defined message – but a reboot does not take place.
Can you confirm this behaviour? Is this intended?
Thanks,
Holger
George Spiers
April 18, 2017Hi Holger
Are you using 7.12? There is an issue in 7.12 where reboots do not process if a notification message is set. Citrix released a hotfix (LC6766) and the issue is resolved in 7.13
Holger
April 19, 2017Yes, I’m using 7.12 and I knew that there is an issue with reboots – but I thought that only it only appears when using the GUI or the old powershell scripts.
Thank you!
Holger
Karthik
May 8, 2017Hello
Can we create Custom reboot schedule, afetr 14 days machine to be rebooted.Please update can we update the same in restart policy.
George Spiers
May 8, 2017No you can do a maximum of one week.
Sarj
June 28, 2017Do you need to create separate reboot schedules for each delivery group or can one schedule be applied to all delivery groups and managed via tag?
George Spiers
June 28, 2017Yes you need separate schedules for each DG.
Srini
August 3, 2017Hi George,
I would like to create weekly reboot schedules (one for each day), will I be able to use the reboot schedules across multiple delivery groups or they are restricted to single delivery group?
George Spiers
August 3, 2017You need separate reboot schedules for each Delivery Group, so no sharing is allowed.
Jon
December 12, 2017It’s pretty silly that its either Daily, or Weekly..and you cannot add more than 1 day to a schedule. why does Citrix insist on moving backwards when they release new products. Here we are in 7.15 and Studio still cannot interact w the V2 commands.
Suchit Kumar
January 15, 2018Can we make 15 days reboot for Tag Servers.
George Spiers
January 16, 2018No you can only reboot once per week at a maximum.
Terry
January 30, 2018Great article, I have this working in my test environment, however in prod it is not running.
Test environment: All components at 7.15 on VMware 6.0 – Working for 3 groups via tags.
Prod environment: VDA is 7.12 with DC , SF and PVS all at 7.15, VMware hypervisor at 5.5.
Is it possible the v2 of the power shell commands do not work with the older hypervisor?
George Spiers
January 31, 2018If vSphere is on v5.5 update 3 then it is supported with XenDesktop 7.12 and 7.15.
If you are using notification messages in your reboot schedules then you will need to upgrade the VDAs past 7.12, alternatively there is a private fix available from Citrix.
Terry
January 31, 2018Thank you George! Do you know the fix name or kb? I will request it or have our Architect get it for us.
George Spiers
January 31, 2018Sure it is LC6766.
Prab
February 25, 2018Hi George, If I am creating 2 reboot schedules for 200 VMs:
> Saturday – for 100 VMs
> Sunday – For rest 100 VMs
My Uses on one of the saturday VMs get a warning to logoff else he will lose his works. He logs off and logs in again, how will be ascertained the does not connect back to the same machine which is about to be rebooted?
George Spiers
February 25, 2018Hi Prab once a VDA is on course to be rebooted, the VDA does not accept any new connections until it has been restarted.
Anonymous
March 6, 2018Got it. Thanks George!
Anonymous
March 16, 2018Hi George if I set a reboot schedule @3:00 am so with a 5 minute frequency it means machine first message will appear @2:45 am and then @2:50 am and @2:55
George Spiers
March 16, 2018Yes but that depends on what value you have assigned for “Warning duration” and “Reboot duration”.
Simon Magee
March 27, 2018I am going to give this a go, I have a customer environment (7.15 LTSR) and a scheduled reboot every day @ 2am with restart all servers in 30 minutes. The next morning one of the servers will be powered off, which has been powered down at the time of the scheduled reboot. This way I can stagger them using Tags and see how I get on.
Rob Treweek
April 18, 2018If you have Proof of Concept or a tiny environment with the Delivery Controller and VDA components on same server, do Restart Schedules still work?
George Spiers
April 18, 2018I honestly have not tried that but that would be unwanted behaviour anyway, unless you are of course only testing the concept of scheduled reboots.
Anonymous
April 20, 2018Thanks George. Just did some testing and the VDA will only restart through the schedule once the Controller role has been removed.
Derek
April 21, 2018Is there a way to force the reboots even with user’s logged in? We have some ‘phantom’ sessions w/ 7.12 VDA that prevent the reboot from occurring.
George Spiers
April 21, 2018Using this method there is no way to force as such. You would have to deploy an additional script for example to force reboot if the scheduled restart failed. I would try and figure out why you have sessions that will not log off and that prevent a reboot.
Niki
June 11, 2018Implemented your set up and noticed not all of my XenDesktops register after the reboot. Noticed they reboot 3 times then go into maintenance mode then get powered off. Why?
George Spiers
June 12, 2018That is not normal, paste the output of Get-BrokerRebootScheduleV2 also make sure the Delivery Group has no v1 reboot schedule set against it.
Kev
June 18, 2018Will this work in 7.15 CU2 ? Also can this be applied to Windows 10 VDA?
George Spiers
June 18, 2018Windows Desktops don’t use reboot schedules. Non-persistent VMs by default reboot once a user logs off.
Brandon
August 23, 2018What time zone is the reboot schedule processed under? The time zone of the DDC? or the time zone of the logged in user? We have offshore users in India and users at our corp office in the US central time zone. If we set a 12AM to 1AM reboot schedule … is it for the time zone of the DDC (in our case the DDC is in the central time zone) or the time zone of the logged in user? The delivery group we would have the schedule set for is shared by onshore and offshore workers (about a 90/10 split onshore/offshore). We are worried that if we set a 12AM reboot schedule based on central time in our home office where the DDC is located that that offshore users (10 hours ahead of us) would be rebooted between 10AM and 11AM in their time (which is of course business hours for them).
George Spiers
August 24, 2018It will be done with the time of the DDC.
Rich
September 7, 2018Hey George,
Great article. Just wanted to see if you had any ideas for my scenario since this changes from 6.5.
Today in 6.5
We reboot a set of servers each weeknight. We use a worker group, load evaluator, and policy for each weeknight. So we have 5 of each. The servers are taken out of load at 6:00 PM, reboot at 3:00 AM, and are put back into load at 5:00 AM each night. As you can see we allow a long drain period as our users are shift based and typically work 8 hours or less so this way we limit any disruptions.
Tomorrow in 7.15 CU2
There doesnt seem to be a clean way to limit disruptions due to the “phases” of the reboot cycle. The only way I can see us getting as close as we have it today in 6.5 is the following:
Scheduled task and powershell script to put servers into maintenance mode at 6:00 PM and out of maintenance mode at 3:00 AM, allowing all servers that will reboot that night to begin draining users. Set a reboot cycle to begin at 3:00 AM with a reboot duration of 0 and a warning duration of 15 min. But will setting the reboot duration to 0 reboot all the servers at once and create a boot storm or will it still do the 2 phases of server reboots?
If I understand the 7.x reboot cycle and phases correctly, then there is a chance that when we take the servers out of maintenance mode at 3:00 AM, the servers in Phase 2 will start allowing logins and thereby a chance that someone will land on those servers but then get disrupted shortly after logging in. Is that correct? If so, do you see any other method to limiting disruptions rathen than minimizing them?
George Spiers
September 9, 2018Hi Rich
The only problem with putting a VDA in Maintenance Mode is that Delivery Controllers will not reboot VDAs in Maintenance Mode, so you cannot depend on Studio reboot schedules. You will have to script the reboots aswell, or else allow Studio to place the VMs in drain mode whenever a reboot schedule is due to commence.
To answer your question a reboot duration of 0 will reboot all machines part of that reboot schedule at once, so I would not take these VDAs out of Maintenance Mode until you are sure they have all been rebooted (the process has started) in this scenario.
You might find that scripting will offer you more chance to get things how you want.
Imran
September 27, 2018Hi Georgie.
Is there any limit to assign servers per delivery group.. Can I have 1300 servers in a single DG
George Spiers
September 27, 2018Yes, I have witnessed more in a single DG.
Josh
October 4, 2018Hey Georgie,
I have 1000+ Servers in 1 DG, Iam having trouble in configuring reboot schedule. Sometimes servers are getting stuck while rebooting or missing schedule. Can you suggest best way how to effectively reboot..
George Spiers
October 8, 2018I use tags, so VMs reboot at slightly different times for me, but if I was you I would check the Delivery Controller event logs for errors, and make sure your DDCs and VDAs are running at the same level and in support. You can also script the reboots using PowerShell, which would achieve “in-guest” rebooting rather than having the DDCs via APIs reboot the VMs for you.
Josh
October 8, 2018Thanks for you inputs. I checked even log 3104 and I don’t find any information about reboot details for that DG, however those machines are getting restarted. And from powershell I executed get-rebootcycle I found that particular DG state is abandoned.. any other clients how to fix this??
George Spiers
October 11, 2018What version of XA/XD are you running? Does it work if no notification message is sent?
Have you both a v1 and v2 restart schedule created for the same Delivery Group?
Try running the below commands using PowerShell:
Remove-BrokerRebootSchedule –DesktopGroupName “YourDGName”
Remove-BrokerRebootScheduleV2 –DesktopGroupName “YourDGName”
After that, re-create the schedule.
Josh
October 11, 20187.15
Tou
October 24, 2018Hi,
We have Worker(s) who reboot every 2 days, the reboot is done over a period of 4 hours.
Sometimes a vda is in “Power Off” status
Do you have any idea?
George Spiers
October 24, 2018Not sure why. Anything in Event Logs of DDC? Have you tried deleting and recreating the reboot schedule?
Tou
October 29, 2018no I haven’t tried to recreate the restart schedule yet?
In the Citrix forums I was advised to change this setting
Set-BrokerDesktopGroup -Name -PeakBufferSizePercent 100 -OffPeakBufferSizePercent 100
Rikesh
November 5, 2018Hi
What is the powershell command to restart Citrix Desktop to revert back to template?
George Spiers
November 5, 2018The PowerShell command to delete the v2 restart schedule?
Anonymous
December 6, 2018Hi,
Great article George. I have created a script using your blog and have tested as well, everything is working fine. Please help me with a particular requirement from the customer.
Is there a way I can enforce end users not to login to the servers which are part of reboot schedule. Suppose, if the server has to start at 3:00AM, no new users shall be able to login to these server. I checked around and found ProhibitNewLogOnsUntilRestart option but this option is available only for 7.6 and above. My XenApp version is 7.15.
Can you please help me with this.
George Spiers
December 6, 2018As part of your script you could place the VDAs in maintenance mode:
Set-BrokerMachineMaintenanceMode -InputObject DOMAIN\CitrixDesktop1 $true
You can then take the machines out of maintenance mode after they have rebooted.
Phil
March 14, 2019We’re on 7.15 CU3 and running into a weird issue which some other people have alluded to. We have around 40 servers reboot nightly and every morning a few random devices are powered off. After looking through all of the logs I think what’s happening is the devices are originally getting their power off and turn on commands BUT after that they either aren’t coming back up fully or are getting hung. As a result they’re not registering so the DDC sends another power off command to them. Any suggestions on how to resolve this?
George Spiers
March 15, 2019Check the Delivery Controller event logs for power actions from DDCs. They will confirm if what you think is happening actually is happening. If it is what is happening, and consistently some virtual machines are always powered off each morning, you need to catch the virtual machines in the act to better understand what is going wrong.
Phil
March 18, 2019We’re using VMware (6.5) as our Hypervisor. In checking out the logs it’s actually a little different than I thought. The DDC is sending a ‘shutdown’ command first, which is the VMware equivalent of “initiate guest OS shutdown.” That is where the VM is getting hung, its not fully powering off. After several minutes there’s a new event saying the VM could not be powered down and then the DDC sends a ‘turnoff’ command, which is the VMware force power off. After that, the VM is powered off but there’s no addition ‘turnon’ or ‘power on’ command sent from the DDC. I’ve opened a case with Citrix on it.
George Spiers
March 18, 2019That appears to be normal behaviour. If there is no registration in 20 minutes a hard power off occurs. The problem is understanding why some VMs do not shut down properly.
Anonymous
April 15, 2019Yeah, citrix pointed us to these three articles:
https://support.citrix.com/article/CTX218117
https://support.citrix.com/article/CTX214838
https://support.citrix.com/article/CTX239537
We actually ended up setting the following keys which did end up powering on the VDAs after a forced power off :
[HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\DesktopServer\RebootSchedule]
“MaxShutdownDelayMin”
“ShutdownTimeoutRecovery”
“MaxRegistrationDelayMin”
However, with other limitations for Citrix’s Reboot Schedule I think we may move over to a custom script and move away from their solution. Thanks!
Phil
April 15, 2019Yeah, citrix ended up pointing us to these three articles:
https://support.citrix.com/article/CTX218117
https://support.citrix.com/article/CTX214838
https://support.citrix.com/article/CTX239537
We ended up setting the following registry keys and the VDAs did receive a follow-up power on command and came up successfully:
[HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\DesktopServer\RebootSchedule]
“MaxShutdownDelayMin”
“ShutdownTimeoutRecovery”
“MaxRegistrationDelayMin”
However, due to other limitations with Citrix’s Reboot Schedule I think we may move away from it and use other custom scripting. Thanks!
David
January 31, 2020Would there be a way to Reboot with by two tags like above but for every other day for each group like HR (Mon_Wed_Fri) IT( Tue_Thur_Sat)
George Spiers
May 10, 2020That should work fine yes.
Pawel
April 15, 2020Hello,
Please tell me if it’s possible to set scheduler for PM or in 24-hour clock ?
Best Regards
George Spiers
September 7, 2020You can use the 24-hour time format.
sudhir
April 30, 2020i want to reboot my VDAS from Monday to Friday on daily basis. dont want my machines to reboot on sat and sunday.
Any script or suggestions please
Anonymous
April 30, 2020my env is XA 7.15
George Spiers
September 9, 2020Five weekly schedules, one for each working day.
Nishith Gupta
July 22, 2020In 1912 LTSR, reboot schedules configured through xxx-BrokerRebootScheduleV2 show up in Studio.
Sahil Aaron
May 9, 2021In 1912 LTSR, Still there is issue with reboot of Server VDA which has Notification enabled. It is not yet solved.
TimC
September 3, 2021I want Saturday not Sunday. I dont see where you specify day
Chris
November 5, 2021What is actually initiating the reboots on the backend once the schedule is set?
Anonymous
March 30, 2022I am also interested in this.
We have an issue with scheduled reboots since the update to 1912 LTSR CU5. Sometimes not all machines of an delivery group are rebooted, but they should have to.
B
May 12, 2022Hello!
Do you have any update / fix? I experience the same issue for 1912 LTSR CU5.
Thank you.
B
May 3, 2022Same issue with scheduled reboots for 1912 LTSR CU5. Only half of the server reboot in a DG. Is there any fix, please? Thank you.
Anonymous
May 30, 2022We also have this issue with scheduled reboots since the update to 1912 LTSR CU5. Sometimes not all machines of an delivery group are rebooted, but they should have to. Is there already a fix?
B
May 30, 2022We’ve opened a citrix case but no fix yet.