Answered by:
Moving Files to folders based on First 4 digits of Name. If Folder doesn't exist DONT move file

Question
-
I'm moving files to folders based on the 1st 4 digits of the folder name.
If the first 4 digits don't correspond to a folder name how do you just ignore the error and dont move the file? I'm thinking its in the OnError Event handler but again, there isnt much in the way of decent documentation on how to do this.
Any help would be appreciated
Debbie
Tuesday, May 28, 2013 2:03 PM
Answers
-
You can add a script task to check if the folder is present beore moving the file using a script task (ST1) and based on the script task's result you can execute the file system task (FST2) to move the files within the foreach loop as ST1-->FST2.
Create 2 variables: Exists (Int) and FolderPath (I think you already have this variable). Take a script task and configure it as:
- Script Language: C#
- Read only variable: FolderPath
- Read write variable: Exists
- Use following code in script task:
using System; using System.Data; using Microsoft.SqlServer.Dts.Runtime; using System.Windows.Forms; using System.IO; namespace ST_58766d3d454f4e008ccd08ee2bd2181e.csproj { [System.AddIn.AddIn("ScriptMain", Version = "1.0", Publisher = "", Description = "")] public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase { #region VSTA generated code enum ScriptResults { Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success, Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure }; #endregion public void Main() { //If folder exists if (Directory.Exists(Dts.Variables["FolderPath"].Value.ToString())) { //Directory.CreateDirectory(Dts.Variables["FolderPath"].Value.ToString(), null); Dts.Variables["Exists"].Value = 1; } else { Dts.Variables["Exists"].Value = 0; } Dts.TaskResult = (int)ScriptResults.Success; } } }
Add a precedence constraint between ST1 and FST2 (double click the thick green arrow between ST1 and FST2) as:
- Evaluation Operation: Expression and Constraint
- Value: Success
- Expression: @[User::Exists]==1
Nitesh Rai- Please mark the post as answered if it answers your question
- Proposed as answer by ArthurZ Tuesday, May 28, 2013 2:47 PM
- Edited by Nitesh Rai Tuesday, May 28, 2013 2:48 PM
- Marked as answer by debznfoz Monday, June 3, 2013 12:02 PM
Tuesday, May 28, 2013 2:46 PM -
Still stuck on this Ill try and make everything a little bit clearer. (For myself and any help I can get)
VARIABLES
I have the following variables
ToSchoolsFullFilePath - Value C:\Services\To_Schools\CAF\4197_CAF_Testfile.xls
ToSchoolsSourcePath – Value C:\Services\To_Schools\
SchoolFromToSchoolsFullFilePath – Expression
SUBSTRING(REVERSE(SUBSTRING(REVERSE(@[User::ToSchoolsFullFilePath]),5,FINDSTRING(REVERSE( @[User::ToSchoolsFullFilePath] ),"\\",1)-5)),1,4)
Which Evaluates to (For Example 4197)
Ensure EvaluateAsExpression is set to True
FOR EACH LOOP
I have a For Each Loop
Collection - ForEach Loop Editor Expressions Directory = @[User::ToSchoolsSourcePath]
Variable Mapping @[User::ToSchoolsFullFilePath]
FILE SYSTEM TASK
And I have a File System Task to move the files to folders dependant upon the 1<sup>st</sup> 4 digits of the code.
IsDestinationPathVariable False
Destination Connection 1101 Which links to the connection Manager 1101 Expression String
"C:\\HotFolders\\Upload\\"+ @[User::SchoolFromToSchoolsFullFilePath]
SourceConnection True User::ToSchoolsFullFilePath
If I run this with the correct file in it works perfectly. All the Files in Services folder in Upload got to the corresponding 4 digit schools folders.
If I add another file that has no corresponding folder (Which may happen)
It errors [File System Task] Error: An error occurred with the following error message: "Could not find a part of the path.".
So for this I have added the new developments
Variables
HotFoldersUploadPath – Value C:\HotFolders\Upload
HotFoldersUploadPathWithSchool - Expression @[User::HotFoldersUploadPath]+ "\\"+@[User::SchoolFromToSchoolsFullFilePath]
If you remember SchoolFromToSchoolsFullFilePath is an original variable and evaluates to the 4 digit School Code
I have also set the property EvaluateAsExpression = True
This full variable evaluates to C:\HotFolders\Upload\4197
Exists int 1 or 0
Script
ReadOnlyVariables User::HotFoldersUploadPathWithSchool
ReadWriteVariables User::Exists
[CODE]
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Runtime;
using System.Windows.Forms;
using System.IO;
namespace ST_58766d3d454f4e008ccd08ee2bd2181e.csproj
{
[System.AddIn.AddIn("ScriptMain", Version = "1.0", Publisher = "", Description = "")]
public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
{
#region VSTA generated code
enum ScriptResults
{
Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
};
#endregion
public void Main()
{
//If folder exists
if (Directory.Exists(Dts.Variables["HotFoldersUploadPathWithSchool"].Value.ToString()))
{
//Directory.CreateDirectory(Dts.Variables["FolderPath"].Value.ToString(), null);
Dts.Variables["Exists"].Value = 1;
}
else
{
Dts.Variables["Exists"].Value = 0;
}
Dts.TaskResult = (int)ScriptResults.Success;
}
}
}
[/CODE]
This script task sits within the ForEach loop before the Script task and is joined by a green arrow.
On the green arrow pointing to the script task
Expression and Constraint
Value Success
Expression @[User::Exists]==1
Logical AND
So the directory MUST exist or else the file is passed over.
I run this and everything runs APART from the script task. It seems that everything is being set as 0 and I cant quite figure out why.
The variable HotFoldersUploadPathWithSchool Must be the problem. Having another look at this
It evaluates to what I need C:\HotFolders\Upload\4197
So Again I’m at a loss as to why this isn’t working?
Any help on this would be greatly appreciated. Ive come to a bit of a halt on this.
I was hoping to find a way to just keep going if a file fails without having to use a script because clearly Im struggling.
Debbie
- Marked as answer by debznfoz Monday, June 3, 2013 12:03 PM
Monday, June 3, 2013 10:03 AM
All replies
-
Hi Debbie,
According to your earlier question you wanted to create the folders based on the file names (first 4 characters of filename). If that is the case then there is no case of folder not existing.
Nitesh Rai- Please mark the post as answered if it answers your question
Tuesday, May 28, 2013 2:13 PM -
There has been a spec change. I wanted to add the folders based on the folder names and then add the files.
However I've been told that every code should already be set as a folder name before hand. (There are 420) Which means if a User Messes up and Adds a file with an incorrect file name, the file doesnt get moved along with all the files with codes that match the folders.
(Im hoping to create some kind of text file log of the files that dont get moved so I can email the users and say you need to sort your file out. Its not been uploaded. but I'm way off getting to this point)
Tuesday, May 28, 2013 2:25 PM -
It means that folders are already created/present and you need to move the files to their respective folders. In case there is no folder available for a file, it should skip that file. Is that correct?
Nitesh Rai- Please mark the post as answered if it answers your question
Tuesday, May 28, 2013 2:28 PM -
Thats right!
At the moment it errors because "Part of the filepath is missing"
I want it to just carry on moving the files, which will just leave me with any files that dont match the folders
Tuesday, May 28, 2013 2:31 PM -
You can add a script task to check if the folder is present beore moving the file using a script task (ST1) and based on the script task's result you can execute the file system task (FST2) to move the files within the foreach loop as ST1-->FST2.
Create 2 variables: Exists (Int) and FolderPath (I think you already have this variable). Take a script task and configure it as:
- Script Language: C#
- Read only variable: FolderPath
- Read write variable: Exists
- Use following code in script task:
using System; using System.Data; using Microsoft.SqlServer.Dts.Runtime; using System.Windows.Forms; using System.IO; namespace ST_58766d3d454f4e008ccd08ee2bd2181e.csproj { [System.AddIn.AddIn("ScriptMain", Version = "1.0", Publisher = "", Description = "")] public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase { #region VSTA generated code enum ScriptResults { Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success, Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure }; #endregion public void Main() { //If folder exists if (Directory.Exists(Dts.Variables["FolderPath"].Value.ToString())) { //Directory.CreateDirectory(Dts.Variables["FolderPath"].Value.ToString(), null); Dts.Variables["Exists"].Value = 1; } else { Dts.Variables["Exists"].Value = 0; } Dts.TaskResult = (int)ScriptResults.Success; } } }
Add a precedence constraint between ST1 and FST2 (double click the thick green arrow between ST1 and FST2) as:
- Evaluation Operation: Expression and Constraint
- Value: Success
- Expression: @[User::Exists]==1
Nitesh Rai- Please mark the post as answered if it answers your question
- Proposed as answer by ArthurZ Tuesday, May 28, 2013 2:47 PM
- Edited by Nitesh Rai Tuesday, May 28, 2013 2:48 PM
- Marked as answer by debznfoz Monday, June 3, 2013 12:02 PM
Tuesday, May 28, 2013 2:46 PM -
Ooooh Some C# Script!
I will attempt to get my head round this! Thank you again
Debbie
Tuesday, May 28, 2013 2:48 PM -
Hi Debbie,
Much of the code is aut generated and code you want to look into is the if-else logic in Main function. It checks if a folder exists or not and sets a package variable with 1 or 0.
Nitesh Rai- Please mark the post as answered if it answers your question
Tuesday, May 28, 2013 3:39 PM -
Ive got my head around the code I think and Ive done all the instructions.
If the folder Exists set the exists value to 1. If it doesnt set to 0.
Then only the ones where exists = 1 are moved in the next loop because its used in the expression for success on the green arrow
//If folder exists
if (Directory.Exists(Dts.Variables["FolderPath"].Value.ToString()))
{
//Directory.CreateDirectory(Dts.Variables["FolderPath"].Value.ToString(), null);
Dts.Variables["Exists"].Value = 1;
}
else
{
Dts.Variables["Exists"].Value = 0;However I am getting this error message, which is because of the Exists variable I have tried int16, int32 and int64 but all 3 produce the same error
SSIS package "Package.dtsx" starting.
Error: 0xC001F009 at Package: The type of the value being assigned to variable "User::Exists" differs from the current variable type. Variables may not change type during execution. Variable types are strict, except for variables of type Object.
Error: 0x1 at Script Task: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> Microsoft.SqlServer.Dts.Runtime.DtsRuntimeException: The type of the value being assigned to variable "User::Exists" differs from the current variable type. Variables may not change type during execution. Variable types are strict, except for variables of type Object.
---> System.Runtime.InteropServices.COMException (0xC001F009): The type of the value being assigned to variable "User::Exists" differs from the current variable type. Variables may not change type during execution. Variable types are strict, except for variables of type Object.
at Microsoft.SqlServer.Dts.Runtime.Wrapper.IDTSVariable100.set_Value(Object pvValue)
at Microsoft.SqlServer.Dts.Runtime.Variable.set_Value(Object value)
--- End of inner exception stack trace ---
at Microsoft.SqlServer.Dts.Runtime.Variable.set_Value(Object value)
at ST_58766d3d454f4e008ccd08ee2bd2181e.csproj.ScriptMain.Main()
--- End of inner exception stack trace ---
at System.RuntimeMethodHandle._InvokeMethodFast(Object target, Object[] arguments, SignatureStruct& sig, MethodAttributes methodAttributes, RuntimeTypeHandle typeOwner)
at System.RuntimeMethodHandle.InvokeMethodFast(Object target, Object[] arguments, Signature sig, MethodAttributes methodAttributes, RuntimeTypeHandle typeOwner)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture, Boolean skipVisibilityChecks)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at System.RuntimeType.InvokeMember(String name, BindingFlags bindingFlags, Binder binder, Object target, Object[] providedArgs, ParameterModifier[] modifiers, CultureInfo culture, String[] namedParams)
at System.Type.InvokeMember(String name, BindingFlags invokeAttr, Binder binder, Object target, Object[] args, CultureInfo culture)
at Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTATaskScriptingEngine.ExecuteScript()
Task failed: Script Task
Warning: 0x80019002 at Package: SSIS Warning Code DTS_W_MAXIMUMERRORCOUNTREACHED. The Execution method succeeded, but the number of errors raised (3) reached the maximum allowed (1); resulting in failure. This occurs when the number of errors reaches the number specified in MaximumErrorCount. Change the MaximumErrorCount or fix the errors.
SSIS package "Package.dtsx" finished: Failure.
Wednesday, May 29, 2013 9:17 AM -
Can you check the scope of the variable Exists? Also, have you added Exists as read/write variable?
Nitesh Rai- Please mark the post as answered if it answers your question
Wednesday, May 29, 2013 9:29 AM -
I hvae added a variable Exists Data Type Int32 (Currently)
In the ScriptTask in ReadWriteVariables User::Exists
On the green arrow I have for the expression @[User::Exists]==1
Im not sure what you mean by check the scope of the variable, is that something to do with checking that the naming conventions all match?
Wednesday, May 29, 2013 9:42 AM -
When you create a variable using the variable window, there is a column Scope (second column; just after Name column. This scope column shows the scope of the variable. The value for this column should be the name of the package).
Nitesh Rai- Please mark the post as answered if it answers your question
Wednesday, May 29, 2013 10:09 AM -
Ah of course, The scope for all of my variables is 'package' Ive made sure they are all set at the top level so they can be used for everything.
Well thats strange i just tried it again without changing everything and It succeeded. its now erroring at the moving files section. Its erroring with the original error message
[File System Task] Error: An error occurred with the following error message: "Could not find a part of the path.".
its still trying to send the (Test) file through that Ive created with a code that isnt yet a folder, Which is really good news, at least its succeeding but its not doing what I want yet. It the variable thats being used thats at fault.
Im using Variables["ToSchoolsSourcePath"] in place of your original folderpath and i have added the value for the school path up to the 4 day code.
Which is why its not working. So I have changed it to the variable of HotFoldersUploadPathWithSchool
And I have set an expression up for it of @[User::HotFoldersUploadPath]+ "\\"+@[User::SchoolFromToSchoolsFullFilePath]
Which evaluates as C:\HotFolders\Upload\4197
Which should work because this changes every time you run through the files. This is now getting away frm me a bit. I can understand within the loop to move the files that its traversing subfolders to look at files in these sub folders. I dont see the corresponding information here. I dont really know what its doing.
Wednesday, May 29, 2013 10:41 AM -
Ahhhh They should probably be both in the loop, thats where Im getting confused. Ill have a rethink!Wednesday, May 29, 2013 10:58 AM
-
Ahhhh They should probably be both in the loop, thats where Im getting confused. Ill have a rethink!
By "both", if you mean script task and file system task, then yes. Both should be within for each loop containerNitesh Rai- Please mark the post as answered if it answers your question
Wednesday, May 29, 2013 11:06 AM -
SO VERY CLOSE, but beginning to get so complicated I can barely see whats happening
It now runs the script but doesnt do anything else (It doesnt move on to the file task)
From the first forum post about moving files based on 4 digits of the file name
The Variable mapping @User::ToSchoolsFullFilepath I have another value e.g. C:................ToSchools\CAF\4197_CAF_Example.xls
Then before the move file task I have placed the script task inside this.
ReadOnly @User::HotFoldersUploadPathWithSchool Created as a variable of @[User::HotFoldersUploadPath]+ "\\"+@[User::SchoolFromToSchoolsFullFilePath]
Which evaluates as C:................\HotFolders\Upload\4197
This variable is used inside the script its self
So the two variable that create this are User::HotFoldersUploadPath value of C:..............\Upload
AND User::SchoolFromToSchoolsFullFilePath Created with a variable of SUBSTRING(REVERSE(SUBSTRING(REVERSE(@[User::ToSchoolsFullFilePath]),5,FINDSTRING(REVERSE( @[User::ToSchoolsFullFilePath] ),"\\",1)-5)),1,4)
And this just creates the Code, e.g. 4197
So I would expect that its going through all the files, finding the 4 digit code and checking they match in the Upload folder. If not its a 0. If yes it can continue.
But again its getting so complicated, Im really begining to struggle with keeping a hold on the logic.
Wednesday, May 29, 2013 12:07 PM -
Still stuck on this Ill try and make everything a little bit clearer. (For myself and any help I can get)
VARIABLES
I have the following variables
ToSchoolsFullFilePath - Value C:\Services\To_Schools\CAF\4197_CAF_Testfile.xls
ToSchoolsSourcePath – Value C:\Services\To_Schools\
SchoolFromToSchoolsFullFilePath – Expression
SUBSTRING(REVERSE(SUBSTRING(REVERSE(@[User::ToSchoolsFullFilePath]),5,FINDSTRING(REVERSE( @[User::ToSchoolsFullFilePath] ),"\\",1)-5)),1,4)
Which Evaluates to (For Example 4197)
Ensure EvaluateAsExpression is set to True
FOR EACH LOOP
I have a For Each Loop
Collection - ForEach Loop Editor Expressions Directory = @[User::ToSchoolsSourcePath]
Variable Mapping @[User::ToSchoolsFullFilePath]
FILE SYSTEM TASK
And I have a File System Task to move the files to folders dependant upon the 1<sup>st</sup> 4 digits of the code.
IsDestinationPathVariable False
Destination Connection 1101 Which links to the connection Manager 1101 Expression String
"C:\\HotFolders\\Upload\\"+ @[User::SchoolFromToSchoolsFullFilePath]
SourceConnection True User::ToSchoolsFullFilePath
If I run this with the correct file in it works perfectly. All the Files in Services folder in Upload got to the corresponding 4 digit schools folders.
If I add another file that has no corresponding folder (Which may happen)
It errors [File System Task] Error: An error occurred with the following error message: "Could not find a part of the path.".
So for this I have added the new developments
Variables
HotFoldersUploadPath – Value C:\HotFolders\Upload
HotFoldersUploadPathWithSchool - Expression @[User::HotFoldersUploadPath]+ "\\"+@[User::SchoolFromToSchoolsFullFilePath]
If you remember SchoolFromToSchoolsFullFilePath is an original variable and evaluates to the 4 digit School Code
I have also set the property EvaluateAsExpression = True
This full variable evaluates to C:\HotFolders\Upload\4197
Exists int 1 or 0
Script
ReadOnlyVariables User::HotFoldersUploadPathWithSchool
ReadWriteVariables User::Exists
[CODE]
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Runtime;
using System.Windows.Forms;
using System.IO;
namespace ST_58766d3d454f4e008ccd08ee2bd2181e.csproj
{
[System.AddIn.AddIn("ScriptMain", Version = "1.0", Publisher = "", Description = "")]
public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
{
#region VSTA generated code
enum ScriptResults
{
Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
};
#endregion
public void Main()
{
//If folder exists
if (Directory.Exists(Dts.Variables["HotFoldersUploadPathWithSchool"].Value.ToString()))
{
//Directory.CreateDirectory(Dts.Variables["FolderPath"].Value.ToString(), null);
Dts.Variables["Exists"].Value = 1;
}
else
{
Dts.Variables["Exists"].Value = 0;
}
Dts.TaskResult = (int)ScriptResults.Success;
}
}
}
[/CODE]
This script task sits within the ForEach loop before the Script task and is joined by a green arrow.
On the green arrow pointing to the script task
Expression and Constraint
Value Success
Expression @[User::Exists]==1
Logical AND
So the directory MUST exist or else the file is passed over.
I run this and everything runs APART from the script task. It seems that everything is being set as 0 and I cant quite figure out why.
The variable HotFoldersUploadPathWithSchool Must be the problem. Having another look at this
It evaluates to what I need C:\HotFolders\Upload\4197
So Again I’m at a loss as to why this isn’t working?
Any help on this would be greatly appreciated. Ive come to a bit of a halt on this.
I was hoping to find a way to just keep going if a file fails without having to use a script because clearly Im struggling.
Debbie
- Marked as answer by debznfoz Monday, June 3, 2013 12:03 PM
Monday, June 3, 2013 10:03 AM -
Hi Debbie,
If I understand it correctly, you are able to move the files to their respective folders (in case folders exists) but if the folder does not exist then file system task fails despite script task checking the existence of the folder before executing the file system task.
You said " ....I run this and everything runs APART from the script task...."
Do you mean that script task is not executed? Can you post a screen shot of the foreach loop?
Script task should run first and then the file system task.
Nitesh Rai- Please mark the post as answered if it answers your question
Monday, June 3, 2013 10:38 AM -
well THATS odd
I have a Forloop that deletes the Files in the Upload Folder befor we start. If I run everything with this delete files the Move Files doesnt change colour meaning it doesnt run (Within the fol loop.
However If I run just the Forloop it works. Strange behiour. I wouldnt expect the delete files section to cause problems at the next stage???
Monday, June 3, 2013 11:28 AM -
Well Ive changed the Destination in the file system task to Overwrite destination = True and got rid of the delete files task to make it work. I cant for the list of me figure out why the delete causes it not to run though??? very strange
Debbie
Monday, June 3, 2013 11:35 AM -
Delete file will remove the file itself while overwriting a file is simply to overwrite the contents of the file.
Nitesh Rai- Please mark the post as answered if it answers your question
Monday, June 3, 2013 11:42 AM -
Absoloutely, I Understand that, but I dont understand why having a forEach loop at the start of the process (As above) which deleted everything in the destination folders, would cause the newxt forEach Loop not to run to completion.
E.g. the actual move from source to destination doesnt run?
Well its working in a fashion now but this has confused me.
Monday, June 3, 2013 11:46 AM -
If second foreach loop doesn't get any matching file name (as per it's configuration) in ToSchoolIsSourcePath folder (or sub folders) then it will not execute the inner tasks.
Nitesh Rai- Please mark the post as answered if it answers your question
Monday, June 3, 2013 11:50 AM -
It would do through because the files are being cleared from the destination NOT from the source.
Monday, June 3, 2013 12:04 PM