FileSieve 4

Example: Classification Method not finding any matches if no items are processed.

The problem appears to be related to there being items to be processed, but they have all been set to Item.IsProcessed = False. This means that no items have been processed, which seems to cause a problem with the Windows Taskbar progress indicator.

Fixing this will have to be an ongoing process; in the meantime, EnableTaskbarProgress has been added to Settings - Advanced and should be set to False should the issue of the FileSieve window staying disabled after processing has completed.

The Tag plugin is where the issue was found; the EXIF plugin will be investigated later.

The Tag plugin doesn’t actually rename processed files, but instead just creates directories based on the entered Name Mask.

Was this the original intention? Doesn’t seem anywhere near as useful as also being able to rename the files.

For example, original file:

• my_song.mp3

Name Mask: %ALBUM_TITLE%\%TITLE%.%FILE_EXTENSION%

This results in:

• My Album\My Song.mp3\my_song.mp3

It should be:

• My Album\My Song.mp3

The original filename for the file doesn’t get changed by FileSieve when it gets processed. FileSieve doesn’t seem to modify the filename at all.

That can’t be right?

Changing a Modifier’s settings, Okaying the Modifier window, and reopening it shows the changed settings as expected; but restarting the application shows the original settings before the save.

Certain files seem to be ignored by either the engine scanner or somewhere else upstream.

I can’t seem to reproduce this with initial testing.

This also makes me think the Review Settings section should have a Debug/Verbose mode that contains additional information for cases such as these.

2 BIN files, 2 IND files, 2 BDM files, 1 XML file and 2 files with no
extension do not get copied even though .* is set and and minimum filesize is 0

Thanks to Olivier C. for reporting

This may be down to the Windows API call FindNextFile() and file handles.

While it may not actually be a problem, I’d rather not use more memory the more files that are processed - even if memory usage goes up by just a few hundred megabytes for 4 to 5 million files, eliminating this memory usage would still be a win.

Allow the re-ordering of Nodes rather than just allowing Nodes to have their Parent changed.

There are conflicts between mnemonics on the main form due to somewhat forgetting that they need to be unique to the entire form and not just the panel the relevant controls are on.

There are a lot of controls across all the containers, so this’ll need a bit of planning.

When a file is excluded using the Simulation window, the corresponding SourceItem’s listviewitem needs to be updated within the Step 1: Source Items listview so the Ignore column is synched.

The internal regular expression engine in .NET uses the dollar symbol for referencing captures groups; change FileSieve so it uses that character and invoke the .NET Regex.Replace() method instead of using custom code.

Rather than just indicating what should happen to source items via the Copy Mode, plugins need to have the ultimate say in what happens to items/files during processing.

For example, the Archive Modifier compresses items into the destination location and the original file needs to either ignored (if Copy Mode is set to Copy), or deleted (if Copy Mode is set to Move).

The plugin(s) will then need to say “Okay, ignore/delete the original file but count it as being processed”.

The Delete functionality is implemented via the Item.DeletionMode property that is set via plugins, but this more than likely needs to be rolled into a general ProcessMode property to incorporate different types of outcomes that the plugin requires and FileSieve knows about.

no task description

Threaded pre-scanning is partially in-place, but an effective thread initialiser is halting things. It needs to be intelligent so that it can determine the correct number of threads to be used in all cases, from number of files to if files reside on the same or different physical media.

This has the potential for enormous speed gains.

Being able to reference capture groups would be very useful.

Or add a checkbox to the Source Item advanced settings that says “Do not perform a pre-scan” as it’s not actually mutually-exclusive with the current Get Files Mode.

This needs to be thought out better from a code point-of-view.

Leaving for now.

The CopyFileEx copying code was written for something else but it would be perfect for FileSieve as it gives information on the status of the current copy progress and supports checksum verification for copied files.

Must support copying to/from network paths.

Moving files to destination mirrors would probably be more useful than just copying them, but this will require significant core changes that it’ll take a while to implement.

Need to tread carefully as a lot of stuff could get broken.

The UI for some plugin editors are a bit all-over-the-place and need cleaning up.

The UIs for regular expressions need some thought as the concept is confusing to users that have never used it before and FileSieve needs to offer as much help with them as possible. Also, regex capture groups will need to be implemented and the UI should account for that/offer as much help as possible.

After processing, show detailed information on what happened. For example, the Skipped Items stat doesn’t actually give you useful information on what was actually skipped. The Log is a pain in the arse for determining that information (if at all; had a quick look but couldn’t find anything, so don’t know if it actually says what was skipped or not!).

Give full info on what was skipped, errored, etc - and why.

When re-running the same profile against a folder that has already been processed, it would show the number of files that has been added in the last run.

Suggested by Olivier C.

Convert all literal strings over to BiQubic.Suite.AppFramework‘s translation system, which also needs completing.

Ensure there isn’t any locations where an error can bring down the entire application.

no task description

Add a Best Fit that will attempt to place files into groups of a certain filesize.

See Knapsack Problem.

A live preview of the current settings while they’re being edited.

FileSieve wasn’t designed with this capability in mind, and so this would need to be thought out and tested at every step - multiple times.

The population code has been made modular in preparation for this addition.

no task description

The Method and Modifier labels on the Options tab should be colour-coded depending on the status of the respective Method and Modifier.

If no Method is set, then colour the Method label red. If a Method is set but it doesn’t have valid/sane settings, then colour the label orange-ish.

Ditto for the Modifiers label, except for the red = no Modifier set as that’s valid.

Make another pass over the entire FileSieve interface to ensure tooltips are present and comprehensive.

This will be a FileSieve v4 update of the v3 plugin, with additional features.

If possible, extend this to Modifiers. Custom Modifers scripts was originally intended, but I can’t seem to remember why that’s a no-go.

Pressing Escape in any non-code editor should give focus to the code editor. This is VERY important and why I hate most code editors.

no task description

This won’t be as easy as implementing a traditional plugin as it will possibly require core changes that touches lots of areas.

Take information out of files and use that information to create directories.

This will no-doubt need to be text-based.

A database of filetypes that the Item Editor can use.

The database will need to be exposed via the API so other plugins can make use of it, such as the Classification plugin.

Maybe create a web-service that FileSieve can connect to to update its database, but that’s more than likely overkill as it’s doubtful I’ll have any other applications that can make use of this information.

Allow the archiving of processed items.

To make the Enterprise Edition more suited for an enterprise environment, add support for storing settings/profiles/etc on a remote server that FileSieve can sync to that data.

Add support for the BiQubic.Bugs exception library.

Once it has been rewritten, anyway. On top of that, should it submit exceptions to this tracker instead of its own site? Not really sure.

Allow the user to add specific profiles to the Windows Explorer right-click shell context menu so profiles can be quickly run on a directory.

“Shell codes” (eg. %shell_file1%) will need to be added in various parts of FileSieve so Explorer folders/files can be referenced and acted upon.

Add some Profile presets.

Allow Start Points and End Points to be specified.

For example, setting a Start Point of HTTP will allow FileSieve to process items from a HTTP server and setting an End Point of FTP will allow FileSieve to use an FTP server as a destination.

This will probably require introducing an intermediate “language” that describes what FileSieve is trying to do and is interpreted by the respective Start/End Points into concrete actions.

This’ll allow Start/End Points of other services such as Drop Box, Google Drive, Amazon S3, etc.

Definitely requires some planning.

Add more commands to the Automation system; such as specifying Methods, Modifiers, etc.

New processing modes that allow the returning of just paths, as opposed to files.

This will make developing Methods simpler if they only care about directories, such as the Size method that has been waiting to be developed for the past year.

New Method that allows the processing of files based on their contents.

Add additional overwrite codes to the Overwrite Rules Editor, such as %created_year% (original file’s created date year), %created_month%, and so on.

When the Overwrite Rules Editor was originally implemented, FileSieve didn’t collect comprehensive metadata on the files it processed, such as file dates and the like. It’s been a while since metadata collection was added, so the rules editor needs to be updated to support that information.

IMPORTANT CHANGES TO SOME EXISTING CODES

 %file% is now %file_name%
 %ext% is now %file_ext%
 # is now %file_number%

In the Step 5 Processing: Add a line: ‘Added Items’. When re-running the same profile against a folder that has already been processed, it would show the number of files that have been added in the last run.

Thanks to Olivier C. for suggesting

There will be additional settings available that apply to the overwrite rules, specifically how FileSieve should determine what constitutes an ‘identical’ file. For example, compare just file names; or file names, date stamps; or hashing; or all of those.

The Destination area of FileSieve simply isn’t big enough to contain these extra options - and it would look like a mess even if it was - so a new window will be added to contain them.

If a detailed-enough log is kept of a full “processing”, then maybe this’ll allow the possibility of “undoing” a processing after-the-fact?

Would need proper planning and checks in place as it could get somewhat messy if anything unexpected happens.

Add a ton of additional formatting options and configurability(?) such as back in the days of Elixir. Maybe put such options within an Advanced section or something, to keep the main interface clean.

Create a new Method that determines what to do based on a rule set created by the user.

If example, you can add an IF rule that will perform a check and if it’s True, it’ll continue running through the rest of the rules. Or maybe it’ll allow other arbitrary things to be performed.

Not too easy to explain, but imagine a wizard but geared more towards logic.

The most amount of work here is designing the interface.

The Trigger system should be consolidated into the Profile Manager, and any future features - such as Thread Settings - can easily be added without cluttering up the main window UI.