Bug in TFS 2017 Collection Upgrade?

Question

hi, we are doing a "practice upgrade" from TFS 2015 RTM to TFS 2017, (for the second time), and seem to be running into a bug that crashes a collection upgrade.   Bear in mind that there has only been time, so far to reproduce this issue twice, and I will describe what seems to be the steps to reproduce it.

In both run-throughs, (using different collections, but getting the same error in the log when the crash occurred), it only occurred on the _first_ collection upgrade, and I suspect that you have to follow a particular pattern of upgrading in order to get the crash.    Since we always follow this pattern it has happened to us twice now.

First, establish a TFS 2015 RTM server with a few collections.   When you upgrade to 2017, follow this pattern:

While on the old software detach all collections.  Remove the application tier.

Install TFS 2017.  

Configure the application Tier and let 2017 upgrade your configuration database.

Attach the first collection.   Around steps 22-30 it has crashed with the following error at the tail end of the log:

[13:13:45.863] +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
[13:13:45.863] Executing step: Add the User Extension License for the attached collection
[13:13:45.863]   Executing step: 'Add the User Extension License for the attached collection' ExtensionLicensing.CopyUserExtensionLicenseToConfigDb (693 of 994)
[13:13:45.880]   Original Server Id: 1d9e3916-d1e4-4295-9bdd-2327b213e27b, Current Server Id : 1d9e3916-d1e4-4295-9bdd-2327b213e27b
[13:13:45.880]   Copy user extension license data back to configuration database from snapshot
[13:13:45.897]   Create temp user extension license table for attach
[13:13:46.100]   Target partition Id: 1, Collection Id: d69b9ea1-a591-44d2-adb9-c622ab8e2b2c
[13:13:46.113]   Getting tables. Table filter: tbl_UserExtensionLicense
[13:13:46.113]   Getting indexes. Table filter: tbl_UserExtensionLicense
[13:13:46.520]   Found 0 indexes. Table filter: tbl_UserExtensionLicense
[13:13:46.757]   Found 0 tables. Table filter: tbl_UserExtensionLicense
[13:13:46.757]   Getting tables. Table filter: tbl_TempUserExtensionLicense
[13:13:46.757]   Getting indexes. Table filter: tbl_TempUserExtensionLicense
[13:13:47.413]   Found 0 indexes. Table filter: tbl_TempUserExtensionLicense
[13:13:47.627]   Found 1 tables. Table filter: tbl_TempUserExtensionLicense
[13:13:47.643]   [Error] Object reference not set to an instance of an object.
[13:13:47.660]   System.NullReferenceException: Object reference not set to an instance of an object.
[13:13:47.660]      at Microsoft.VisualStudio.Services.Licensing.OnPremPlugins.ExtensionLicensingStepPerformer.BulkCopyUserExtensionLicense(ServicingContext servicingContext, IVssRequestContext sourceRequestContext, IVssRequestContext targetRequestContext, Int32 targetPartitionId, Guid collectionId, String targetTableName)
[13:13:47.660]      at Microsoft.VisualStudio.Services.Licensing.OnPremPlugins.ExtensionLicensingStepPerformer.CopyUserExtensionLicenseToConfigDb(IVssRequestContext targetRequestContext, ServicingContext servicingContext)
[13:13:47.660]      at Microsoft.TeamFoundation.Framework.Server.TeamFoundationStepPerformerBase.PerformHostStep(String servicingOperation, ServicingOperationTarget target, IServicingStep servicingStep, String stepData, ServicingContext servicingContext)
[13:13:47.660]      at Microsoft.TeamFoundation.Framework.Server.TeamFoundationStepPerformerBase.PerformStep(String servicingOperation, ServicingOperationTarget target, IServicingStep servicingStep, String stepType, String stepData, ServicingContext servicingContext)
[13:13:47.660]      at Microsoft.TeamFoundation.Framework.Server.TeamFoundationStepPerformerBase.PerformStep(String servicingOperation, ServicingOperationTarget target, String stepType, String stepData, ServicingContext servicingContext)
[13:13:47.660]      at Microsoft.TeamFoundation.Framework.Server.ServicingStepDriver.PerformServicingStep(ServicingStep step, ServicingContext servicingContext, ServicingStepGroup group, ServicingOperation servicingOperation, Int32 stepNumber, Int32 totalSteps)
[13:13:47.660] Step failed: Add the User Extension License for the attached collection. Execution time: 1797 milliseconds.
[13:13:47.660]   [StepDuration] 1.7979332
[13:13:47.673]   [GroupDuration] 40.3609754
[13:13:47.673]   [OperationDuration] 40.867523
[13:13:47.753]   Clearing dictionary, removing all items.

In the first run-through, I restored the collection from backup, and created a new TFS_Configuration database.   Then when I attached, the collection upgraded with no issue.

I  have just discovered that all of my collections are crashing with this same error no matter how many times I restore the databases and attempt the upgrade.   We are in trouble.   At present I know of no way to get around this anymore other than possibly to recreate the TFS_Configuration database.

I'll have to try that to see if it works.   However, it leaves me unsure about possible permissions issues that might become unset as a result of doing this.

Any help would be appreciated.

Can you give me any insights or work-arounds on this issue?

Regards,

Steve Sims

Answer 1

Some data, in case this helps to provide insight:

Our 'old' server was running Windows Server 2012 R2.   Database was running on SQL Server Enterprise 2014.   TFS version was 2015 RTM.

The new server, which is failing the collection upgrades with the above error is Windows Server 2016, and SQL Server 2016 Standard.

Consider and let me know if this issue is being caused somehow by the data compression feature of SQL Server Enterprise 2014 not really being compatible with the data compression in SQL Server 2016 Standard.   From what I have read this should not be the issue, but I wanted to mention it.

Answer 2

Steve,

if you install SQL 2016 SP1 than most features from Enterprise edition are now available in Standard Edition too, e.g. data compression, partitioning etc.

But I don't believe that this is a problem because you can't attach a database to a lower edition if it does use features from the higher edition - eg. data compression.

The licensing had changed several times during TFS 2015 updates cycle and I would recommand you to first upgrade to TFS 2015 Update 3 and than check upgrade to TFS 2017.

are you using the same domain and the same service accounts for TFS 2015 and TFS 2017 ?

have you added extensions to TFS 2015?

---

Please use Mark as Answer if my post solved your problem and use Vote As Helpful if a post was useful.

Answer 3

hi Daniel,

Thank you for the reply.   We are doing a bit better at this moment.

First, I already installed SQL Server SP1.   Without that, our backups don't event restore to SQL (even before any TFS involvement).   Without SP1, the restore operation does climb through 100% and then suddenly we get a "restore failed" message with the database left in "suspect" state.    SP1 corrected this behavior.

As it turns out, the issue with the TFS collection upgrade _is_ as you suspect, being caused by a change in licensing.   I got my "practice session media" by downloading it from: https://www.visualstudio.com/en-us/news/releasenotes/tfs2017-relnotes

However, I installed from our original media first, and upgraded from there.   But while after the initial installation we were showing License Type as "Volume Licensing", this disappeared once we did the final upgrade to 2017 using this media as downloaded.

Keep in mind that in our practice scenario, we are trying to follow exactly what we are going to do on the go-live upgrade.   Therefore, as we begin it, we followed the "Upgrade production" installation path rather than the "New Deployment" path which would have created a new Tfs_Configuration database.  

The thing is, the "licensing wizard" which contains the button to obtain a "free" license is only reached when you follow the "New Deployment" wizard path, so I never saw it.   One thing I wondered was that if I could have been given a free license, and the software recognized I didn't have one, could it either have alerted me, or else just given me a free one by default behavior.

Also, you would think that if you embark on a many hours upgrade that Microsoft would put up a warning and say: “Whoa there hoss, you don’t have a license.   Go to this menu and get one free.”   But no, in fact they perform the many hours upgrade and fail it at the end after all the time has elapsed, and then really don’t clearly tell you why leaving you to think it is a bug maybe.

I have definitely lost from 24 to 36 hours over this lack of clarity in how this is organized.

Once I decided to try recreating the Tfs_Configuration database instead of upgrading our existing one, I found the button and pressed it.   Then restored databases and upgraded.   One collection has now upgraded with no errors, and the second one is 13 hours into it.

I will be using the new Tfs_Configuration database merely to perform the collection upgrades.   Once they are upgraded I will detach the upgraded collections, restore our original Tfs_Configuration database and attach them to that.

Oh well.

We will definitely have the right media when we go-live.   It was a rather painful experience, but I do appreciate your input.   If you can communicate this experience it would probably be helpful for your organization to have these insights.

Steve Sims

Answer 4

I will be using the new Tfs_Configuration database merely to perform the collection upgrades.   Once they are upgraded I will detach the upgraded collections, restore our original Tfs_Configuration database and attach them to that.

what are you trying here to do ?

if you want to upgrade the existing server from TFS 2015 to TFS 2017 why do you not an in-place upgrade ? (after taking a full SQL Server database backup)

---

Please use Mark as Answer if my post solved your problem and use Vote As Helpful if a post was useful.

Answer 5

Do you mean upgrade the Tfs_Configuration database and all collection databases in one single operation?

If that's what you mean, then the reason is that such a single operation would take something like 30 hours.   Since we have, in the past experienced failures in either our procedures or Microsoft software issues, we like to have a more incremental process, rather than to wait 30 hours for a single pass / fail operation that could stand to waste so much time.

Answer 6

Correction, I have just calculated that the upgrade of our biggest collection will take just under 28 hours by itself, which is the 30 hours I mentioned.   Including taking backups, restorations, switching machines, and so forth, the entire upgrade process will take in excess of 50 hours, and that is just for the production server machine per se.    Because of the configuration changes we have something like 27 build machines on which to upgrade the configuration for SSL.   Our initial upgrade for the main build agents to vNext, of course is separate and those agents must be separately upgraded, not to mention script changes for all of our builds that are already vNext to take advantage of the features that were missing on 2015 RTM, such as our work around to get tfs labels, and convert the gated builds.   Etc.

There are other related changes that can wait past the main upgrade, but the above mentioned changes all have to happen "at once" to keep the development organization "invisible" to the downtime.   So the entire upgrade process will entail something like 65 hours or so.

Answer 7

Hi Steve,
I am sorry you are having difficulties with upgrade to TFS 2017. The issues you are seeing has nothing to with TFS licensing. BTW: Aaron has a good blog post about licensing changes: https://blogs.msdn.microsoft.com/visualstudioalm/2016/04/06/team-foundation-server-update-2-trial/

What you are doing is rather unusual: detach collections, upgrade config db, attach collections.

Recommendation from TFS product team is to upgrade all databases in 1 shot without doing detach/attach.

Some customers are creating new TFS instance with new config db and then do detach/attach dance.

When collection is detached from the TFS 2017, we create Licensing.tbl_UserExtensionLicense table in the collection database and copy rows that are applicable to this collection from the same table in the config db. In attach code, we copy data back to the configuration database if we detect that collection is attached to the same server. Unfortunately, the code does not take into account the fact that collections detached from TFS 2015 and earlier do not have this table. We will fix this issue in TFS 2017 Update 1.

To workaround it, you can execute the following statement in the detached collection database before attach (or after attach failed):

IF (SCHEMA_ID(N'Licensing') IS NULL) EXEC (N'CREATE SCHEMA Licensing AUTHORIZATION dbo')
GO

IF OBJECT_ID('Licensing.tbl_UserExtensionLicense', 'U') IS NULL
BEGIN
    CREATE TABLE Licensing.tbl_UserExtensionLicense(
	    PartitionId INT NOT NULL,
	    InternalScopeId INT NOT NULL,
	    UserId UNIQUEIDENTIFIER NOT NULL,
	    ExtensionId VARCHAR(200) NOT NULL,
	    Source TINYINT NOT NULL,
	    Status TINYINT NOT NULL,
	    CollectionId UNIQUEIDENTIFIER NULL,
	    AssignmentDate DATETIME NOT NULL,
	    LastUpdated DATETIME NULL
    )
END

I am surprised that upgrade from TFS 2015 to 2017 will take 28 hours for you. How did you came up with this number? Can you share upgrade log via email? (vladimir.khvostov at microsoft.com)

Daniel,

unfortunately, TFS 2017 RTM will not take advantage of premium features like page compression available in SQL Server 2016 SP1 in Standard and Express Editions. We are planning to make necessary changes in TFS 2017 Update 1 or Update 2.

Thanks,

--Vladimir

Answer 8

I had to switch Microsoft accounts, but I am the same OP.

Vladimir, thank you for your reply.

My numbers for the amount of time to upgrade came from an actual practice upgrade sessions using the actual (then current) database backups on test servers.

However, I do know of an issue that may have elongated the times for the collection upgrades.

Our existing production server is a Hyper-V vm running Server 2012 R2.   The test upgrade was done on a Hyper-V vm running Windows Server 2016.

Our Hyper-V machine settings on the old machine include the hyper-v option "Use Dynamic Memory".    We replicated this on the new (Windows Server 2016) machine.    Something about the "Use Dynamic Memory" feature of hyper-v did not work right with Windows Server 2016.    The effect was rather odd.   On the performance tab (memory) of task manager, merely refreshing the TFS Admin Console caused this to shoot up from nearly no memory in use to nearly all of the memory in the machine being used.

However, clicking on the details tab did not reveal any difference in any running processes memory usage.

I went through closing, or stopping services, or killing as many processes as I could (as a test), but the performance tab still showed almost all memory in use.   Only rebooting the machine brought it back down to normal......and that only lasted until the TFS Admin console was refreshed, which shot it back up again.

However, once we turned off the "Use Dynamic Memory" feature of Hyper-V, this effect went away, and the machine began acting just as normally as the old machine.   We had no further problems after that.

But our long test collection upgrades occurred before we even discovered the issue.   I theorize that SQL server was being choked out of sufficient memory, and was struggling along during the whole of our test upgrade.   That might explain why we had elongated times.

Answer 9

Vladimir,

Based on your recommendation and some checking with the information you provided I am going to redo our entire practice upgrade test following your recommendations.    Based on the outcome we may alter our plans for the go-live upgrade on this coming Friday.

I checked the table that you mentioned above, watching it create in the collection when the detach occurs, and examining the copy in the Configuration database.    On the practice server, both are empty (i.e. have no rows in them).

I will restore the original 2015 databases on the practice server and try the method of creating that table, and attaching.   After that test, I will also restore all of them and do a one-shot upgrade (which will take some time.)

I am expecting that by following your recommendations I will have rows in that table.   I am thinking that the table(s) are empty because of the detach/attach method of upgrading.

Just to clarify, we prefer the detach/attach method because of potential issues with hardware, power fail and so forth like the one I described above.  Since we have multiple collections, if we experience a failure during the second or later collection upgrade it allows us to rollback to just after the last successful collection upgrade and continue.    With a one-shot, we have no such means of doing that.    If a one-shot upgrade fails at any point, we have to start completely over again.

Steve

Answer 10

hi Vladimir,

I have completed a one-shot practice session.   There were no issues and the upgrade did go faster.

However, looking at the Licensing.tbl_UserExtensionLicense table , there are still no contents in that table, even when the upgrade is done as a one-shot upgrade.

I can detect no difference between running the upgrade as a one-shot, or a detach/attach operation.

When I use the proper media downloaded from MSDN, I never get the error shown by the logs above.   I am a little worried about the lack of any rows in the Licensing.tbl_UserExtensionLicense table, but that is the same regardless of how I upgrade.

Looking at the 2017 Admin Console after the upgrade, and clicking on the server level (first line at the top), I no longer see a product ID, or any License Type information.   (Again, this is true even when the upgrade is a one-shot upgrade.)

Steve

Answer 11

hi Vladimir,

I have finally realized that the table we are talking about is for TFS User Extensions (duh!) and we don't have any.   That's why the table has no rows.

I think that with your help above, we are all good to go now.

Thanks!

Steve