A working Test Track Pro Adapter for the TFS Integration Platform

well, it has been a long road from misery to hope with a little disbelief thrown in for good measure, but i finally have a working adapter for the tfs integration platform.

acknowledgements

• jose luis soria teruel – for his excellent advice and some sample code. i only used some of his code, but knowing that it can be done is the first step to achieving the goal.

updates

• 2011-06-06 11:00 – i found a last minute bug where by the adapter thinks that a work item that was created before the high water mark but was not in scope before it was edited was converted to an “edit” change action instead of a “add”. i updated lines 89 and 102 of the source. the result is a workitemhistorynotfound conflict
  

  [06/06/2011 10:12:14] migrationconsole.exe information: 0 : workitemtracking: processing changegroup #3214, change 3143:2
  [06/06/2011 10:12:15] migrationconsole.exe information: 0 : workitemtracking: unresolved conflict:
  [06/06/2011 10:12:15]   session: adea805d-51df-489a-b2fd-9717b4af3703
  [06/06/2011 10:12:15]   source: 6e3bdf70-f1ae-4cd5-8ee4-133c8aee0857
  [06/06/2011 10:12:15]   message: cannot find applicable resolution rule.
  [06/06/2011 10:12:15]   conflict type: tfs wit history not found conflict type
  [06/06/2011 10:12:15]   conflict type reference name: 1722df87-ab61-4ad0-8b41-531d3d804089
  [06/06/2011 10:12:15]   conflict details: <?xml version="1.0"?>
  [06/06/2011 10:12:15] <workitemhistorynotfoundconflicttypedetails xmlns:xsi="http://www.w3.org/2001/xmlschema-instance" xmlns:xsd="http://www.w3.org/2001/xmlschema">
  [06/06/2011 10:12:15]   <sourceworkitemid>3143</sourceworkitemid>
  [06/06/2011 10:12:15]   <sourceworkitemrevision>2</sourceworkitemrevision>
  [06/06/2011 10:12:15]   <sourcemigrationsourceid>c513f930-2602-400d-a0bf-a2a3ab434df5</sourcemigrationsourceid>
  [06/06/2011 10:12:15]   <targetmigrationsourceid>6e3bdf70-f1ae-4cd5-8ee4-133c8aee0857</targetmigrationsourceid>
  [06/06/2011 10:12:15] </workitemhistorynotfoundconflicttypedetails> 

with the new code, which has gone through many refactors for the sake of last ditch efforts to figure out the bug i am now able to update tfs from ttp in an incremental fashion.

figure: work items are now being updated

imports microsoft.teamfoundation.migration.toolkit
imports system.componentmodel.design
imports system.collections.objectmodel
imports microsoft.teamfoundation.migration.toolkit.services
imports microsoft.teamfoundation.migration.businessmodel
imports microsoft.teamfoundation.migration.toolkit.errormanagement
imports system.globalization
imports system.xml
imports microsoft.teamfoundation.migration.toolkit.syncorchestrator
imports system.net
imports system.io
imports system.servicemodel
imports system.servicemodel.security
imports northwestcadence.ttptipadapter.ttpsoapsdk
imports northwestcadence.ttptipadapter.ttpsoapsdk.api

public class ttpanalysisprovider
    inherits analysisproviderbase

    ' fields
    private _analysisservicecontainer as iservicecontainer
    private _changegroupservice as changegroupservice
    private _configurationservice as configurationservice
    private _conflictmanagerservice as conflictmanager
    private _highwatermarkdelta as highwatermark(of datetime)
    private _highwatermarkchangeset as highwatermark(of integer)
    private _supportedchangeactions as dictionary(of guid, changeactionhandler)
    private _supportedcontenttypes as collection(of contenttype)
    private _datasourceconfig as ttpmigrationdatasource
    private _highwatermarkrevisions as new dictionary(of string, highwatermark(of integer))
    private _tstart as datetime = now

    ' properties
    public overrides readonly property supportedchangeactions as dictionary(of guid, changeactionhandler)
        get
            return me._supportedchangeactions
        end get
    end property

    public overrides readonly property supportedcontenttypes as collection(of contenttype)
        get
            return me._supportedcontenttypes
        end get
    end property

    private shared function createfieldrevisiondescriptiondoc(row as ttpdefectmigrationitem) as xmldocument
        dim columns as new xelement("columns", new object() {new xelement("column", new object() {new xattribute("displayname", "author"), new xattribute("referencename", "author"), new xattribute("type", "string"), new xelement("value", row.authorid)}), new xelement("column", new object() {new xattribute("displayname", "displayname"), new xattribute("referencename", "displayname"), new xattribute("type", "string"), new xelement("value", row.displayname)}), new xelement("column", new object() {new xattribute("displayname", "id"), new xattribute("referencename", "id"), new xattribute("type", "string"), new xelement("value", row.id.tostring)})})
        dim column as keyvaluepair(of string, object)
        for each column in row.columns
            if not string.isnullorempty(column.value) then
                columns.add(new xelement("column", new object() {new xattribute("displayname", column.key), new xattribute("referencename", column.key), new xattribute("type", "string"), new xelement("value", column.value)}))
            end if
        next
        dim descriptiondoc as new xelement("workitemchanges", new object() {new xattribute("revision", row.revision), new xattribute("workitemtype", row.woritemtype), new xattribute("author", iif(string.isnullorempty(row.authorid), "", row.authorid)), new xattribute("changedate", row.modifiedon.tostring(cultureinfo.currentculture)), new xattribute("workitemid", row.id.tostring), columns})
        dim doc as new xmldocument
        doc.loadxml(descriptiondoc.tostring)
        return doc
    end function

    private shared function createfieldcolumn(migrationactiondetails as xmldocument, displayname as string, referencename as string, fieldtype as string, value as object, isskippingfield as boolean) as xmlelement
        dim c as xmlelement = migrationactiondetails.createelement("column")
        c.setattribute("displayname", displayname)
        c.setattribute("referencename", referencename)
        c.setattribute("type", fieldtype)
        c.setattribute("isskippingfield", isskippingfield.tostring())
        dim v as xmlelement = migrationactiondetails.createelement("value")
        'object translatedvalue = translatefieldvalue(f, fieldvalue);
        dim translatedvalue as object = value
        if translatedvalue is nothing then
            v.innertext = string.empty
        else
            translatedvalue.tostring()
        end if
        c.appendchild(v)
        return c
    end function

    public overrides sub generatedeltatable()
        _tstart = now
        try
            dim viewname as string = me._configurationservice.filters.item(0).path
            tracemanager.traceinformation("ttpwit:ap:generatedeltatable:view - {0}", new object() {viewname})
            me._highwatermarkdelta.reload()
            '------------------------------------------------------
            dim context as ttpcontext = getttpcontext()
            dim raw as list(of ttpdefectmigrationitem) = getttprawdata(context, viewname)
            tracemanager.traceinformation("located {0} raw updates since {1} in {2} seconds", raw.count, _highwatermarkdelta.value, now.subtract(_tstart).totalseconds)
            ' find all of the data that needs to be added in this run
            dim deltanew = (from ri in raw where ri.createdon.compareto(_highwatermarkdelta.value) > 0 or ri.revision = 0).tolist
            ' get any extra data and create the add changesets
            tracemanager.traceinformation("located {0} deltas as new in {1} seconds", deltanew.count, now.subtract(_tstart).totalseconds)
            deltanew = getdeltaworkflow(context, deltanew)
            tracemanager.traceinformation("updated {0} deltas with workflow in {1} seconds", deltanew.count, now.subtract(_tstart).totalseconds)
            dim changesnew as list(of changegroup) = getchangegroupsforadds(deltanew)
            tracemanager.traceinformation("created {0} add change groups in {1} seconds", changesnew.count, now.subtract(_tstart).totalseconds)
            ' save the chnagesets to the backing store
            for each c in changesnew
                c.save()
            next
            tracemanager.traceinformation("saved {0} add change groups in {1} seconds", changesnew.count, now.subtract(_tstart).totalseconds)
            ' find all of the data that needs to be edited in this run
            dim deltaedit = (from ri in raw where (ri.modifiedon.compareto(_highwatermarkdelta.value) > 0 and not ri.createdon.compareto(_highwatermarkdelta.value) > 0) or ri.revision > 0).tolist
            ' get any extra data and create the edit changesets
            tracemanager.traceinformation("located {0} deltas as edit in {1} seconds", deltaedit.count, now.subtract(_tstart).totalseconds)
            deltaedit = getdeltaworkflow(context, deltaedit)
            tracemanager.traceinformation("updated {0} deltas with workflow in {1} seconds", deltaedit.count, now.subtract(_tstart).totalseconds)
            dim changesedit as list(of changegroup) = getchangegroupsforedits(deltaedit)
            tracemanager.traceinformation("created {0} edit change groups in {1} seconds", changesedit.count, now.subtract(_tstart).totalseconds)
            ' save the chnagesets to the backing store
            for each c in changesedit
                c.save()
            next
            tracemanager.traceinformation("saved {0} edit change groups in {1} seconds", changesedit.count, now.subtract(_tstart).totalseconds)
            '------------------------------------------------------
            ' update the high water mark and send the changes through
            me._highwatermarkdelta.update(datetime.now)
            me._changegroupservice.promotedeltatopending()
        catch ex as exception
            tracemanager.traceexception(ex)
        end try
    end sub

    public overrides sub initializeclient()
        tracemanager.traceinformation("ttpwit:ap:initializeclient")
    end sub

    private shared function initializemigrationdatasource() as ttpmigrationdatasource
        return new ttpmigrationdatasource
    end function

    public overrides sub initializeservices(byval analysisservice as iservicecontainer)
        tracemanager.traceinformation("ttpwit:ap:initializeservices")
        if (analysisservice is nothing) then
            throw new argumentnullexception("analysisservice")
        end if
        me._analysisservicecontainer = analysisservice
        me._configurationservice = directcast(analysisservice.getservice(gettype(configurationservice)), configurationservice)
        dim migrationsourceconfiguration as migrationsource = me._configurationservice.migrationsource
        _datasourceconfig = ttpanalysisprovider.initializemigrationdatasource
        dim customsetting as customsetting
        dim username as string = ""
        dim password as string = ""
        dim isworkflowincluded as boolean = true
        dim hwmdateoveride as datetime = datetime.minvalue
        for each customsetting in me._configurationservice.migrationsource.customsettings.customsetting
            if customsetting.settingkey.equals("username", stringcomparison.ordinalignorecase) then
                username = customsetting.settingvalue
            end if
            if customsetting.settingkey.equals("password", stringcomparison.ordinalignorecase) then
                password = customsetting.settingvalue
            end if
            if customsetting.settingkey.equals("overridehwm", stringcomparison.ordinalignorecase) then
                if not datetime.tryparse(customsetting.settingvalue, hwmdateoveride) then
                    throw new invalidcastexception("date is not in the correct format: overridehwm")
                end if
            end if
            if customsetting.settingkey.equals("isworkflowincluded", stringcomparison.ordinalignorecase) then
                if not boolean.tryparse(customsetting.settingvalue, isworkflowincluded) then
                    throw new invalidcastexception("date is not in the correct format: isworkflowincluded")
                end if
            end if
        next

        _datasourceconfig.credentials = new networkcredential(username, password)
        _datasourceconfig.databasename = migrationsourceconfiguration.sourceidentifier
        _datasourceconfig.filtername = iif(migrationsourceconfiguration.serveridentifier = "[enterfiltername]", "", migrationsourceconfiguration.serveridentifier)
        _datasourceconfig.url = migrationsourceconfiguration.serverurl
        _datasourceconfig.isworkflowincluded = isworkflowincluded

        me._supportedcontenttypes = new collection(of contenttype)
        me.supportedcontenttypes.add(wellknowncontenttype.workitem)

        dim handler as new ttpchangeactionhandlers(me)

        me._supportedchangeactions = new dictionary(of guid, changeactionhandler)
        me.supportedchangeactions.add(wellknownchangeactionid.add, new changeactionhandler(addressof handler.basicactionhandler))
        me.supportedchangeactions.add(wellknownchangeactionid.edit, new changeactionhandler(addressof handler.basicactionhandler))
        me.supportedchangeactions.add(wellknownchangeactionid.delete, new changeactionhandler(addressof handler.basicactionhandler))
        me._highwatermarkdelta = new highwatermark(of datetime)("hwmdelta")
        me._highwatermarkchangeset = new highwatermark(of integer)("lastchangeset")
        me._configurationservice.registerhighwatermarkwithsession(me._highwatermarkdelta)
        me._configurationservice.registerhighwatermarkwithsession(me._highwatermarkchangeset)

        if hwmdateoveride > datetime.minvalue then
            _highwatermarkdelta.update(hwmdateoveride)
        end if

        me._changegroupservice = directcast(me._analysisservicecontainer.getservice(gettype(changegroupservice)), changegroupservice)

        me._changegroupservice.registerdefaultsourceserializer(new ttpdefectmigrationitemserializer)
    end sub

    public overrides sub registerconflicttypes(byval conflictmanager as conflictmanager)
        tracemanager.traceinformation("ttpwit:ap:registerconflicttypes")
        me._conflictmanagerservice = directcast(me._analysisservicecontainer.getservice(gettype(conflictmanager)), conflictmanager)
        me._conflictmanagerservice.registerconflicttype(new genericconflicttype)
        me._conflictmanagerservice.registerconflicttype(new ttpgeneralconflicttype, conflictssyncorchoptions.continue)
    end sub

    public overrides sub registersupportedchangeactions(byval changeactionregistrationservice as changeactionregistrationservice)
        tracemanager.traceinformation("ttpwit:ap:registersupportedchangeactions")
        changeactionregistrationservice = directcast(me._analysisservicecontainer.getservice(gettype(changeactionregistrationservice)), changeactionregistrationservice)
        dim supportedchangeaction as keyvaluepair(of guid, changeactionhandler)
        for each supportedchangeaction in me.supportedchangeactions
            dim contenttype as contenttype
            for each contenttype in me.supportedcontenttypes
                changeactionregistrationservice.registerchangeaction(supportedchangeaction.key, contenttype.referencename, supportedchangeaction.value)
            next
        next
    end sub

    public overrides sub registersupportedcontenttypes(contenttyperegistrationservice as microsoft.teamfoundation.migration.toolkit.services.contenttyperegistrationservice)

    end sub

    private function getttpcontext() as ttpcontext
        tracemanager.traceinformation("-getttpcontext")
        dim ttpserver as uri = new uri(string.format("{0}/scripts/ttsoapcgi.exe", _datasourceconfig.url))
        tracemanager.traceinformation(chrw(9) & "-getttpcontext loading ttp  {0}", new object() {ttpserver})
        dim context as ttpcontext = nothing
        try
            context = ttpsoapsdkapi.createcontext(ttpserver, _datasourceconfig.databasename, _datasourceconfig.credentials.username, _datasourceconfig.credentials.password)
            tracemanager.traceinformation("-getttpcontext connected to '{0}' on '{1}' in {2}", context.project.database, ttpserver, now.subtract(_tstart).tofriendly)
        catch ex as exception
            tracemanager.traceexception(ex)
        end try
        return context
    end function

    private function getttprawdata(context as ttpcontext, filter as string) as list(of ttpdefectmigrationitem)
        dim raw as new list(of ttpdefectmigrationitem)
        try
            dim columns as list(of ctablecolumn) = context.getcolumns("defect")
            tracemanager.traceinformation("-getttprawdata '{0}' columns in {1}", columns.count, now.subtract(_tstart).tofriendly)
            tracemanager.traceinformation("-getttprawdata atempting get on all data")
            dim rows as crecordlistsoap = context.getrecords("defect", filter, columns)
            tracemanager.traceinformation("-getttprawdata found {0} records in {1} seconds", rows.records.count, now.subtract(_tstart).totalseconds)
            dim currentrecord as integer = 1
            dim countrecords = rows.records.count
            for each record in rows.records
                ' item has been modified since hwm & before deltra table start time
                try
                    dim defectmi as ttpdefectmigrationitem = ttpdefectmigrationitem.convertcdefecttottpdefectmigrationitem(_configurationservice, columns.toarray, record)
                    '----------------
                    raw.add(defectmi)
                    tracemanager.traceinformation("-getttprawdata  {0} of {1} - '{2}' number '{3}' has loaded in {4} seconds", currentrecord, countrecords, defectmi.woritemtype, defectmi.id, now.subtract(_tstart).totalseconds)
                catch ex as exception
                    tracemanager.traceerror("-getttprawdata  {0} of {1} - '{2}' number '{3}' has {4} processing in {5} seconds", currentrecord, countrecords, "unknown", "unknown", "failed", now.subtract(_tstart).totalseconds)
                    tracemanager.traceexception(ex)
                end try
                currentrecord = currentrecord + 1
            next
        catch ex as exception
            tracemanager.traceexception(ex)
        end try
        return raw
    end function

    private function getdeltaworkflow(context as ttpcontext, byval deltas as list(of ttpdefectmigrationitem)) as list(of ttpdefectmigrationitem)
        try
            dim currentrecord as integer = 1
            dim countrecords = deltas.count
            for each di in deltas
                ' item has been modified since hwm & before deltra table start time
                if _datasourceconfig.isworkflowincluded then
                    di.importdefectdata(context)
                    tracemanager.traceinformation("-getdeltaworkflow  {0} of {1} - '{2}' number '{3}' has {4} processing revision {5} in {6} seconds", currentrecord, countrecords, di.woritemtype, di.id, "updated", di.revision, now.subtract(_tstart).totalseconds)
                else
                    tracemanager.traceinformation("-getdeltaworkflow  {0} of {1} - '{2}' number '{3}' has {4} processing revision {5} in {6} seconds", currentrecord, countrecords, di.woritemtype, di.id, "skipped", di.revision, now.subtract(_tstart).totalseconds)
                end if
                currentrecord = currentrecord + 1
            next
        catch ex as exception
            tracemanager.traceexception(ex)
        end try
        return deltas
    end function

    private function getchangegroupsforadds(byval deltas as list(of ttpdefectmigrationitem)) as list(of changegroup)
        dim changes as new list(of changegroup)
        try
            dim currentrecord as integer = 1
            dim countrecords = deltas.count
            for each delta in deltas
                ' item has been modified since hwm & before deltra table start time
                try
                    ' create and add acction group
                    delta.resetrevision()
                    dim changegroup as changegroup = me._changegroupservice.createchangegroupfordeltatable(string.format("{0}:{1}", delta.id, delta.revision))
                    changegroup.status = changestatus.delta
                    changegroup.owner = nothing
                    changegroup.comment = string.format(cultureinfo.currentculture, "changeset {0}", _highwatermarkchangeset.value)
                    changegroup.changetimeutc = datetime.utcnow
                    changegroup.status = changestatus.delta
                    changegroup.executionorder = 0

                    changegroup.createaction( _
                            wellknownchangeactionid.add, _
                            delta, _
                            delta.id, _
                            _datasourceconfig.databasename, _
                            delta.revision, _
                            " ", _
                            wellknowncontenttype.workitem.referencename, _
                            ttpanalysisprovider.createfieldrevisiondescriptiondoc(delta) _
                            )
                    changes.add(changegroup)
                    _highwatermarkchangeset.update((_highwatermarkchangeset.value + 1))
                    ' done
                catch ex as exception
                    tracemanager.traceerror("-getchangegroups  {0} of {1} - '{2}' number '{3}' has {4} processing in {5} seconds", currentrecord, countrecords, "unknown", "unknown", "failed", now.subtract(_tstart).totalseconds)
                    tracemanager.traceexception(ex)
                end try
                currentrecord = currentrecord + 1
            next
        catch ex as exception
            tracemanager.traceexception(ex)
        end try
        return changes
    end function

    private function getchangegroupsforedits(byval deltas as list(of ttpdefectmigrationitem)) as list(of changegroup)
        dim changes as new list(of changegroup)
        try
            dim currentrecord as integer = 1
            dim countrecords = deltas.count
            for each delta in deltas
                ' item has been modified since hwm & before deltra table start time
                try
                    ' create and add acction group
                    delta.incrementrevision()
                    dim changegroup as changegroup = me._changegroupservice.createchangegroupfordeltatable(string.format("{0}:{1}", delta.id, delta.revision))
                    changegroup.status = changestatus.delta
                    changegroup.owner = nothing
                    changegroup.comment = string.format(cultureinfo.currentculture, "changeset {0}", _highwatermarkchangeset.value)
                    changegroup.changetimeutc = datetime.utcnow
                    changegroup.status = changestatus.delta
                    changegroup.executionorder = 0

                    changegroup.createaction( _
                            wellknownchangeactionid.edit, _
                            delta, _
                            delta.id, _
                            _datasourceconfig.databasename, _
                            delta.revision, _
                            " ", _
                            wellknowncontenttype.workitem.referencename, _
                            ttpanalysisprovider.createfieldrevisiondescriptiondoc(delta) _
                            )
                    changes.add(changegroup)
                    _highwatermarkchangeset.update((_highwatermarkchangeset.value + 1))
                    ' done
                catch ex as exception
                    tracemanager.traceerror("-getchangegroups  {0} of {1} - '{2}' number '{3}' has {4} processing in {5} seconds", currentrecord, countrecords, "unknown", "unknown", "failed", now.subtract(_tstart).totalseconds)
                    tracemanager.traceexception(ex)
                end try
                currentrecord = currentrecord + 1
            next
        catch ex as exception
            tracemanager.traceexception(ex)
        end try
        return changes
    end function

end class

figure: full source for the analysis provider

private function getchangegroupsforedits(byval deltas as list(of ttpdefectmigrationitem)) as list(of changegroup)
    dim changes as new list(of changegroup)
    try
        dim currentrecord as integer = 1
        dim countrecords = deltas.count
        for each delta in deltas
            ' item has been modified since hwm & before deltra table start time
            try
                ' create and add acction group
                delta.incrementrevision()
                dim changegroup as changegroup = me._changegroupservice.createchangegroupfordeltatable(string.format("{0}:{1}", delta.id, delta.revision))
                changegroup.status = changestatus.delta
                changegroup.owner = nothing
                changegroup.comment = string.format(cultureinfo.currentculture, "changeset {0}", _highwatermarkchangeset.value)
                changegroup.changetimeutc = datetime.utcnow
                changegroup.status = changestatus.delta
                changegroup.executionorder = 0

                changegroup.createaction( _
                        wellknownchangeactionid.edit, _
                        delta, _
                        delta.id, _
                        _datasourceconfig.databasename, _
                        delta.revision, _
                        " ", _
                        wellknowncontenttype.workitem.referencename, _
                        ttpanalysisprovider.createfieldrevisiondescriptiondoc(delta) _
                        )
                changes.add(changegroup)
                _highwatermarkchangeset.update((_highwatermarkchangeset.value + 1))
                ' done
            catch ex as exception
                tracemanager.traceerror("-getchangegroups  {0} of {1} - '{2}' number '{3}' has {4} processing in {5} seconds", currentrecord, countrecords, "unknown", "unknown", "failed", now.subtract(_tstart).totalseconds)
                tracemanager.traceexception(ex)
            end try
            currentrecord = currentrecord + 1
        next
    catch ex as exception
        tracemanager.traceexception(ex)
    end try
    return changes
end function

figure: new code to get change groups

i am not exactly positive what made the difference as much of my debugging efforts were hampered by the nasty query bug in ttp , but i am very glad that it is working. it looks like i do not need to have consecutive revision’s although as i have already implemented the code for it i am not going to change it at this stage in the game.

it is now a mater of configuration, but i am creating a table with all of the values of the 120+ fields as well as a neat table for the workflow and inserting it into the history.

figure: loooong history built from ttp data

this history shows all of the values for the fields at the point in time that the data was migrated.

all in, i am quite happy with the process and will be implementing in production really soon. still some testing to do, but all looks good so far.

• can you share your experiences of creating a tfs integration platform adapter?